Business intelligence (bi) werd door Gartner al benoemd tot hoogste prioriteit voor de cio in 2016. Ook de Computable-experts voorspellen dat er veel en grote stappen genomen gaan worden binnen de bi. Tegelijkertijd moeten managers ook terug kijken en nadenken over hun businessmodel bij de inzet van big data: hoe rechtvaardig je de investeringen in big data?

Kurt de Koning, oprichter van Dutch Offshore ICT Management
Business intelligence/analytics is door Gartner op nummer één gezet voor 2016 op de prioriteitenlijst voor de cio. Gebruikers zullen in 2016 hun beslissingen steeds meer laten afhangen van stuurinformatie die uit meerdere bronnen komt. Deze bronnen zullen deels bestaan uit ongestructureerde data. De bi-tools zullen dus niet alleen visueel de informatie aantrekkelijk moeten opmaken en een goede gebruikersinterface moeten bieden. Bij het ontsluiten van de data zullen die tools zich onderscheiden , die in staat zijn om orde en overzicht te scheppen uit de vele verschijningsvormen van data.

Laurent Koelink, senior interim BI professional bij Insight BI
Big data-oplossingen naast traditionele bi
Door de groei van het aantal smart devices hebben organisaties steeds meer data te verwerken. Omdat inzicht (in de breedste zin) een van de belangrijkste succesfactoren van de toekomst gaat zijn voor veel organisaties die flexibel in willen kunnen spelen op de vraag van de markt, zullen zijn ook al deze nieuwe (vormen) van informatie moeten kunnen analyseren. Ik zie big data niet als vervangen van traditionele bi-oplossingen, maar eerder als aanvulling waar het gaat om analytische verwerking van grote hoeveelheden (vooral ongestructureerde) data.

In-memory-oplossingen
Organisaties lopen steeds vaker aan tegen de performance-beperkingen van traditionele database systemen als het gaat om grote hoeveelheden data die ad hoc moeten kunnen worden geanalyseerd. Specifieke hybride database/hardware-oplossingen zoals die van IBM, SAP en TeraData hebben hier altijd oplossingen voor geboden. Daar komen nu steeds vaker ook in-memory-oplossingen bij. Enerzijds omdat deze steeds betaalbaarder en dus toegankelijker worden, anderzijds doordat dit soort oplossingen in de cloud beschikbaar komen, waardoor de kosten hiervan goed in de hand te houden zijn.

Virtual data integration
Daar waar data nu nog vaak fysiek wordt samengevoegd in aparte databases (data warehouses) zal dit, waar mogelijk, worden vervangen door slimme metadata-oplossingen, die (al dan niet met tijdelijke physieke , soms in memory opslag) tijdrovende data extractie en integratie processen overbodig maken.

Agile BI development
Organisaties worden meer en meer genoodzaakt om flexibel mee te bewegen in en met de keten waar ze zich in begeven. Dit betekent dat ook de inzichten om de bedrijfsvoering aan te sturen (de bi-oplossingen) flexibel moeten mee bewegen. Dit vergt een andere manier van ontwikkelen van de bi-ontwikkelteams. Meer en meer zie je dan ook dat methoden als Scrum ook voor bi-ontwikkeling worden toegepast.

Bi voor de iedereen
Daar waar bi toch vooral altijd het domein van organisaties is geweest zie je dat ook consumenten steeds meer en vaker gebruik maken van bi-oplossingen. Bekende voorbeelden zijn inzicht in financiën en energieverbruik. De analyse van inkomsten en uitgaven op de webportal of in de app van je bank, maar ook de analyse van de gegevens van slimme energiemeters zijn hierbij sprekende voorbeelden. Dit zal in de komende jaren alleen maar toenemen en geïntegreerd worden.

Rein Mertens, head of analytical platform bij SAS
Een belangrijke trend die ik tot volwassenheid zie komen in 2016 is ‘streaming analytics’. Vandaag de dag is big data niet meer weg te denken uit onze dagelijkse praktijk. De hoeveelheid data welke per seconde wordt gegenereerd blijft maar toenemen. Zowel in de persoonlijke als zakelijke sfeer. Kijk maar eens naar je dagelijkse gebruik van het internet, e-mails, tweets, blog posts, en overige sociale netwerken. En vanuit de zakelijke kant: klantinteracties, aankopen, customer service calls, promotie via sms/sociale netwerken et cetera.

Een toename van volume, variatie en snelheid van vijf Exabytes per twee dagen wereldwijd. Dit getal is zelfs exclusief data vanuit sensoren, en overige IoT-devices. Er zit vast interessante informatie verstopt in het analyseren van al deze data, maar hoe doe je dat? Een manier is om deze data toegankelijk te maken en op te slaan in een kosteneffectief big data-platform. Onvermijdelijk komt een technologie als Hadoop dan aan de orde, om vervolgens met data visualisatie en geavanceerde analytics aan de gang te gaan om verbanden en inzichten uit die data berg te halen. Je stuurt als het ware de complexe logica naar de data toe. Zonder de data allemaal uit het Hadoop cluster te hoeven halen uiteraard.

Maar wat nu, als je op basis van deze grote hoeveelheden data ‘real-time’ slimme beslissingen zou willen nemen? Je hebt dan geen tijd om de data eerst op te slaan, en vervolgens te gaan analyseren. Nee, je wilt de data in-stream direct kunnen beoordelen, aggregeren, bijhouden, en analyseren, zoals vreemde transactie patronen te detecteren, sentiment in teksten te analyseren en hierop direct actie te ondernemen. Eigenlijk stuur je de data langs de logica! Logica, die in-memory staat en ontwikkeld is om dat heel snel en heel slim te doen. En uiteindelijke resultaten op te slaan. Voorbeelden van meer dan honderdduizend transacties zijn geen uitzondering hier. Per seconde, welteverstaan. Stream it, score it, store it. Dat is streaming analytics!

Minne Sluis, oprichter van Sluis Results
Van IoT (internet of things) naar IoE (internet of everything)
Alles wordt digitaal en connected. Meer nog dan dat we ons zelfs korte tijd geleden konden voorstellen. De toepassing van big data-methodieken en -technieken zal derhalve een nog grotere vlucht nemen.

Roep om adequate Data Governance zal toenemen
Hoewel het in de nieuwe wereld draait om loslaten, vertrouwen/vrijheid geven en co-creatie, zal de roep om beheersbaarheid toch toenemen. Mits vooral aangevlogen vanuit een faciliterende rol en zorgdragend voor meer eenduidigheid en betrouwbaarheid, bepaald geen slechte zaak.

De business impact van big data & data science neemt toe
De impact van big data & data science om business processen, diensten en producten her-uit te vinden, verregaand te digitaliseren (en intelligenter te maken), of in sommige gevallen te elimineren, zal doorzetten.

Consumentisering van analytics zet door
Sterk verbeterde en echt intuïtieve visualisaties, geschraagd door goede meta-modellen, dus data governance, drijft deze ontwikkeling. Democratisering en onafhankelijkheid van derden (anders dan zelfgekozen afgenomen uit de cloud) wordt daarmee steeds meer werkelijkheid.

Big data & data science gaan helemaal doorbreken in de non-profit
De subtiele doelstellingen van de non-profit, zoals verbetering van kwaliteit, (patiënt/cliënt/burger) veiligheid, punctualiteit en toegankelijkheid, vragen om big data toepassingen. Immers, voor die subtiliteit heb je meer goede informatie en dus data, sneller, met meer detail en schakering nodig, dan wat er nu veelal nog uit de traditionelere bi-omgevingen komt. Als de non-profit de broodnodige focus van de profit sector, op ‘winst’ en ‘omzetverbetering’, weet te vertalen naar haar eigen situatie, dan staan succesvolle big data initiatieven om de hoek! Mind you, deze voorspelling geldt uiteraard ook onverkort voor de zorg.

Hans Geurtsen, business intelligence architect data solutions bij Info Support
Van big data naar polyglot persistence
In 2016 hebben we het niet meer over big, maar gewoon over data. Data van allerlei soorten en in allerlei volumes die om verschillende soorten opslag vragen: polyglot persistence. Programmeurs kennen de term polyglot al lang. Een applicatie anno 2015 wordt vaak al in meerdere talen geschreven. Maar ook aan de opslag kant van een applicatie is het niet meer alleen relationeel wat de klok zal slaan. We zullen steeds meer andere soorten databases toepassen in onze data oplossingen, zoals graph databases, document databases, etc. Naast specialisten die alles van één soort database afweten, heb je dan ook generalisten nodig die precies weten welke database zich waarvoor leent.

De doorbraak van het moderne datawarehouse
‘Een polyglot is iemand met een hoge graad van taalbeheersing in verschillende talen’, aldus Wikipedia. Het gaat dan om spreektalen, maar ook in het it-vakgebied, kom je de term steeds vaker tegen. Een applicatie die in meerdere programmeertalen wordt gecodeerd en data in meerdere soorten databases opslaat. Maar ook aan de business intelligence-kant volstaat één taal, één omgeving niet meer. De dagen van het traditionele datawarehouse met een etl-straatje, een centraal datawarehouse en één of twee bi-tools zijn geteld. We zullen nieuwe soorten data-platformen gaan zien waarin allerlei gegevens uit allerlei bronnen toegankelijk worden voor informatiewerkers en data scientists die allerlei tools gebruiken.

Business intelligence in de cloud
Waar vooral Nederlandse bedrijven nog steeds terughoudend zijn waar het de cloud betreft, zie je langzaam maar zeker dat de beweging richting cloud ingezet wordt. Steeds meer bedrijven realiseren zich dat met name security in de cloud vaak beter geregeld is dan dat ze zelf kunnen regelen. Ook cloud leveranciers doen steeds meer om Europese bedrijven naar hun cloud te krijgen. De nieuwe data centra van Microsoft in Duitsland waarbij niet Microsoft maar Deutsche Telekom de controle en toegang tot klantgegevens regelt, is daar een voorbeeld van. 2016 kan wel eens hét jaar worden waarin de cloud écht doorbreekt en waarin we ook in Nederland steeds meer complete BI oplossingen in de cloud zullen gaan zien.

Huub Hillege, principal data(base) management consultant bij Info-Shunt
Big data
De big data-hype zal zich nog zeker voortzetten in 2016 alleen het succes bij de bedrijven is op voorhand niet gegarandeerd. Bedrijven en pas afgestudeerden blijven elkaar gek maken over de toepassing. Het is onbegrijpelijk dat iedereen maar Facebook, Twitter en dergelijke data wil gaan ontsluiten terwijl de data in deze systemen hoogst onbetrouwbaar is. Op elke conferentie vraag ik waar de business case, inclusief baten en lasten is, die alle investeringen rondom big data rechtvaardigen. Zelfs bi-managers van bedrijven moedigen aan om gewoon te beginnen. Dus eigenlijk: achterom kijken naar de data die je hebt of kunt krijgen en onderzoeken of je iets vindt waar je iets aan zou kunnen hebben. Voor mij is dit de grootste valkuil, zoals het ook was met de start van Datawarehouses in 1992. Bedrijven hebben in de huidige omstandigheden beperkt geld. Zuinigheid is geboden.

De analyse van big data moet op de toekomst zijn gericht vanuit een duidelijke business-strategie en een kosten/baten-analyse: welke data heb ik nodig om de toekomst te ondersteunen? Bepaal daarbij:

• Waar wil ik naar toe?
• Welke klantensegmenten wil ik erbij krijgen?
• Gaan we met de huidige klanten meer 'Cross selling' (meer producten) uitvoeren?
• Gaan we stappen ondernemen om onze klanten te behouden (Churn)?

Als deze vragen met prioriteiten zijn vastgelegd moet er een analyse worden gedaan:

• Welke data/sources hebben we hierbij nodig?
• Hebben we zelf de data, zijn er 'gaten' of moeten we externe data inkopen?

Databasemanagementsysteem
Steeds meer databasemanagementsysteem (dbms)-leveranciers gaan ondersteuning geven voor big data-oplossingen zoals bijvoorbeeld Oracle/Sun Big Data Appliance, Teradata/Teradata Aster met ondersteuning voor Hadoop. De dbms-oplossingen zullen op de lange termijn het veld domineren. big data-software-oplossingen zonder dbms zullen het uiteindelijk verliezen.

Steeds minder mensen, ook huidige dbma's, begrijpen niet meer hoe het technisch diep binnen een database/DBMS in elkaar zit. Steeds meer zie je dat fysieke databases uit logische data modelleer-tools worden gegeneerd. Formele fysieke database-stappen/-rapporten blijven achterwege. Ook ontwikkelaars die gebruik maken van etl-tools zoals Informatica, AbInitio, Infosphere, Pentaho et cetera, genereren uiteindelijk sgl-scripts die data van sources naar operationele datastores en/of datawarehouse brengen.

Ook de bi-tools zoals Microstrategy, Business Objects, Tableau et cetera genereren sql-statements.
Meestal zijn dergelijke tools initieel ontwikkeld voor een zeker dbms en al gauw denkt men dat het dan voor alle dbms'en toepasbaar is. Er wordt dan te weinig gebruik gemaakt van specifieke fysieke dbms-kenmerken.

De afwezigheid van de echte kennis veroorzaakt dan performance problemen die in een te laat stadium worden ontdekt. De laatste jaren heb ik door verandering van databaseontwerp/indexen en het herstructureren van complexe/gegenereerde sql-scripts, etl-processen van zes tot acht uur naar één minuut kunnen krijgen en queries die 45 tot 48 uur liepen uiteindelijk naar 35 tot veertig minuten kunnen krijgen.

Advies
De benodigde data zal steeds meer groeien. Vergeet de aanschaf van allerlei hype software pakketten. Zorg dat je zeer grote, goede, technische, Database-/dbms-expertise in huis haalt om de basis van onderen goed in te richten in de kracht van je aanwezige dbms. Dan komt er tijd en geld vrij (je kan met kleinere systemen uit de voeten omdat de basis goed in elkaar zit) om, na een goede business case en ‘proof of concepts’, de juiste tools te selecteren.

3 AI and data science applications that can help dealing with COVID-19

All industries already feel the impact of the current COVID-19 pandemic on the economy. As many businesses had to shut down and either switch to telework or let go of their entire staff, there is no doubt that it will take a long time for the world to recover from this crisis.

Current prospects on the growth of the global economy, shared by different sources, support the idea of the long and painful recovery of the global economy from the COVID-19 crisis.
Statista, for example, compares the initial GDP growth prognosis for 2020 and the prognosis based on the impact of the novel coronavirus on the GPD growth, estimating the difference of as much as 0.5%.

The last time that global GDP experienced such a decline was back in 2008 when the global economic crisis affected every industry with no exceptions.

In the situation with the current pandemic, we also see that different industries change their growth prognoses.
The IT industry, for instance, the expected spending growth in 2020 doesn’t even exceed the pessimistic scenario related to the coronavirus pandemic, and is even expected to shrink.

It would be foolish to claim that the negative effect of the COVID-19 crisis can be reversed. It is already our reality that many businesses and industries around the world will suffer during the current global economic crisis.
Governments around the world responded to this crisis by helping businesses not go bankrupt with state financial support. However, this support is only expected to have a short-term effect and will hardly mitigate the final effect of the global economic crisis on businesses around the world.

So, in search of solutions to decrease the negative effect of drowning global economics, the world, among all other sources, will likely turn to the help of technology, just as the entire world did when it was forced to work from home.

In this article, we offer our stance on how AI and data scientists, in particular, can help respond to the COVID-19 crisis and help relieve its negative effect.

1. Data science and healthcare system

The biggest negative effect on the global economy can come from failing healthcare systems. It was the reason why governments around the world ordered citizens to stay at home and self-isolate, as, in many cases, the course of the COVID-19 disease can be asymptomatic.

Is increasing investment in the healthcare system a bad thing altogether?

No, if we are talking about healthcare systems at a local level, like a state or a province. “At a local level, increasing investments in the healthcare system increases the demand for related products and equipment in direct ratio,” says Dorian Martin, a researcher at WowGrade.

However, in case local governments run out of money in their emergency budgets, they might have to ask the state government for financial support.

This scenario could become our reality if the number of infected people rapidly increases, with hospitals potentially running out of equipment, beds, and, most critically, staff.

What can data science do to help manage this crisis?

UK’s NHS healthcare data storage

Some countries are already preparing for the scenario described above with the help of data scientists.
For instance, the UK government ordered NHS England to develop a data store that would combine multiple data sources and make them deliver information to one secure cloud storage.
What will this data include?

This cloud storage will help NHS healthcare workers access information on the movement of the critical staff, the availability of hospital beds and equipment.

Apart from that, this data storage will help the government to get a comprehensive and accurate view of the current situation to detect anomalies, and make timely decisions based on real data received from hospitals and NHS partner organizations.

Thus, the UK government and NHS are looking into data science to create a system that will help the country tackle the crisis consistently, and manage the supply and demand for critical hospital equipment needed to fight the pandemic.

2. AI’s part in creating the COVID-19 vaccine

Another critical factor that has an effect on the current global economic crisis is the COVID-19 vaccine. It has already become clear that the world is in the standby mode until scientists develop a vaccine that will return people to their normal lives.

It’s a simple cause-and-effect relationship: both global economy and local economies depend on consistent production, production depends on open and functioning production facilities, which depend on workers, who, in their turn, depend on the vaccine to be able to return to work.

And while we still have over a year before the COVID-19 vaccine becomes available to the wide public, scientists turn to AI to speed up the process.

How can AI help develop the COVID-19 vaccine?

• With the help of AI, scientists can analyze the structure of the virus and how it attaches itself to human cells, i.e., its behavior. This data helps researchers build the foundation for vaccine development.
• AI and data science become part of the vaccine development process, as they help scientists analyze thousands of research papers on the matter to make their approach to the vaccine more precise.

An important part of developing a vaccine is analyzing and understanding the protein of the virus and its genetic sequence. In January 2020, Google DeepMind launched a system that builds the virus’s protein in the 3D mode, AlphaFold. This invention already helped the U.S. scientists study the virus enough to create a trial vaccine and launch clinical trials this week.

However, scientists are looking into the ways, how AI can not only be involved in gathering information, but also in the very process of creating a vaccine.

There have already been cases of drugs successfully created by AI. The British startup Excienta created its first drug with the help of artificial intelligence algorithms. The drug is currently undergoing clinical trials. But it will take this drug only 12 months to be ready, compared to 5 years that it usually takes.

Thus, AI gives the world hope that the long-awaited COVID-19 vaccine will be available to the world faster than it’s currently predicted. Yet, there are still a few problems of artificial intelligence implementation in this process, which are mainly connected to AI being underdeveloped itself.

3. Data science and the fight against misinformation

Another factor, which is mostly related to how people respond to the current crisis, and yet has the most negative effect on the global economy, is panic.

We’ve already seen the effects of the rising panic during the Ebola virus crisis in Africa when local economies suffered from plummeting sectors like tourism and commerce.

In economics, the period between the boom (the rising demand for the product) and the bust (a drop in product availability) is very short. During the current pandemic, we’ve seen quite a few examples of how panic buying led to low supply, which damaged local economies.

How can data scientists tackle the threat of panic?

The answer is already in the question: with data.

One of the reasons why people panic is misinformation. “Our online poll has shown that only 12% of respondents read authoritative COVID-19-related resources, while others mostly relied on word-of-mouth approach,” says Martin Harris, a researcher at Studicus.

Misinformation, unfortunately, happens not only among people but on the government level as well. One of the best examples of it is the U.S. officials promoting a drug against malaria as an effective method to treat COVID-19 patients, when, in fact, the effectiveness of this drug hasn’t been proven yet.

The best solution to treat the virus of panic and misinformation is to accumulate all the information from the authoritative resources on the COVID-19 pandemic to help people observe it not only on the local but on the global level as well.

Data scientists and developers at Boston Children’s Hospital have created such a system, called HealthMap, to help people track COVID-19 pandemic, as well as other disease outbreaks around the world.

Conclusion

While there are already quite a few applications of AI and data science that help us respond to the COVID-19 crisis, this crisis is still in its early stages of development.

As we already can use data science to accumulate important information regarding critical hospital staff and equipment, fight misinformation, and use AI to develop the vaccine, we still might discover new ways of applying AI and data science to help the world respond to the COVID-19 crisis.

Yet, today, we can already say that AI and data science have been of enormous help in fighting the pandemic, giving us hope that we will return to our normal lives as soon as possible.

Author: Estelle Liotard

Source: In Data Labs

3 Predicted trends in data analytics for 2021

It’s that time of year again for prognosticating trends and making annual technology predictions. As we move into 2021, there are three trends data analytics professionals should keep their eyes on: OpenAI, optimized big data storage layers, and data exchanges. What ties these three technologies together is the maturation of the data, AI and ML landscapes. Because there already is a lot of conversation surrounding these topics, it is easy to forget that these technologies and capabilities are fairly recent evolutions. Each technology is moving in the same direction -- going from the concept (is something possible?) to putting it into practice in a way that is effective and scalable, offering value to the organization.

I predict that in 2021 we will see these technologies fulfilling the promise they set out to deliver when they were first conceived.

#1: OpenAI and AI’s Ability to Write

OpenAI is a research and deployment company that last year released what they call GPT3 -- artificial intelligence that generates text that mimics text produced by humans. This AI offering can write prose for blog posts, answer questions as a chatbot, or write software code. It’s risen to a level of sophistication where it is getting more difficult to discern if what it generated was written by a human or a robot. Where this type of AI is familiar to people is in writing email messages; Gmail anticipates what the user will write next and offers words or sentence prompts. GPT3 goes further: the user can create a title or designate a topic and GPT3 will write a thousand-word blog post.

This is an inflection point for AI, which, frankly, hasn’t been all that intelligent up to now. Right now, GPT3 is on a slow rollout and is being used primarily by game developers enabling video gamers to play, for example, Dungeons and Dragons without other humans.

Who would benefit from this technology? Anyone who needs content. It will write code. It can design websites. It can produce articles and content. Will it totally replace humans who currently handle these duties? Not yet, but it can offer production value when an organization is short-staffed. As this technology advances, it will cease to feel artificial and will eventually be truly intelligent. It will be everywhere and we’ll be oblivious to it.

#2: Optimized Big Data Storage Layers

Historically, massive amounts of data have been stored in the cloud, on hard drives, or wherever your company holds information for future use. The problem with these systems has been finding the right data when needed. It hasn’t been well optimized, and the adage “like looking for a needle in the haystack” has been an accurate portrayal of the associated difficulties. The bigger the data got, the bigger the haystack got, and the harder it became to find the needle.

In the past year, a number of technologies have emerged, including Iceberg, Hudi, and Delta Lake, that are optimizing the storage of large analytics data sets and making it easier to find that needle. They organize the hay in such a way that you only have to look at a small, segmented area, not the entire data haystack, making the search much more precise.

This is valuable not only because you can access the right data more efficiently, but because it makes the data retrieval process more approachable, allowing for widespread adoption in companies. Traditionally, you had to be a data scientist or engineer and had to know a lot about underlying systems, but these optimized big data storage layers make it more accessible for the average person. This should decrease the time and cost of accessing and using the data.

For example, Iceberg came out of an R&D project at Netflix and is now open source. Netflix generates a lot of data, and if an executive wanted to use that data to predict what the next big hit will be in its programming, it could take three engineers upwards of four weeks to come up with an answer. With these optimized storage layers, you can now get answers faster, and that leads to more specific questions with more efficient answers.

#3: Data Exchanges

Traditionally, data has stayed siloed within an organization and never leaves. It has become clear that another company may have valuable data in their silo that can help your organization offer a better service to your customers. That’s where data exchanges come in. However, to be effective, a data exchange needs a platform that offers transparency, quality, security, and high-level integration.

Going into 2021 data exchanges are emerging as an important component of the data economy, according to research from Eckerson Group. According to this recent report, “A host of companies are launching data marketplaces to facilitate data sharing among data suppliers and consumers. Some are global in nature, hosting a diverse range of data sets, suppliers, and consumers. Others focus on a single industry, functional area (e.g., sales and marketing), or type of data. Still, others sell data exchange platforms to people or companies who want to run their own data marketplace. Cloud data platform providers have the upper hand since they’ve already captured the lion’s share of data consumers who might be interested in sharing data.”

Data exchanges are very much related to the first two focal points we already mentioned, so much so that data exchanges are emerging as a must-have component of any data strategy. Once you can store data more efficiently, you don’t have to worry about adding greater amounts of data, and when you have AI that works intelligently, you want to be able to use the data you have on hand to fill your needs.

We might reach a point where Netflix isn’t just asking the technology what kind of content to produce but the technology starts producing the content. It uses the data it collects through the data exchanges to find out what kind of shows will be in demand in 2022, and then the AI takes care of the rest. It’s the type of data flow that today might seem far-fetched, but that’s the direction we’re headed.

A Final Thought

One technology is about getting access, one is understanding new data, and one is executing information based on the data. As these three technologies begin to mature, we can expect to see a linear growth pattern and see them all intersect at just the right time.

Author: Nick Jordan

Source: TDWI

Investeren in big data betekent het verschil tussen aantrekken of afstoten van klanten, tussen winst of verlies. Veel retailers zien hun initiatieven op het vlak van data en analytics echter doodbloeden. Hoe creëer je daadwerkelijk waarde uit data en voorkom je een opheffingsuitverkoop? Vier tips.

Je investeert veel tijd en geld in big data, exact volgens de boodschap die retailgoeroes al enkele jaren verkondigen. Een team van data scientists ontwikkelt complexe datamodellen, die inderdaad interessante inzichten opleveren. Met kleine ‘proofs of value’ constateert u dat die inzichten daadwerkelijk ten gelde kunnen worden gemaakt. Toch gebeurt dat vervolgens niet. Wat is er aan de hand?

Tip 1: Pas de targets aan

Dat waardevolle inzichten niet in praktijk worden gebracht, heeft vaak te maken met de targets die uw medewerkers hebben meegekregen. Neem als voorbeeld het versturen van mailingen aan klanten. Op basis van bestaande data en klantprofielen kunnen we goed voorspellen hoe vaak en met welke boodschap elke klant moet worden gemaild. En stiekem weet elke marketeer donders goed dat niet elke klant op een dagelijkse email zit te wachten.

Toch trapt menigeen in de valkuil en stuurt telkens weer opnieuw een mailing uit naar het hele klantenbestand. Het resultaat: de interesse van een klant ebt snel weg en de boodschap komt niet langer aan. Waarom doen marketeers dat? Omdat ze louter en alleen worden afgerekend op de omzet die ze genereren, niet op de klanttevredenheid die ze realiseren. Dat nodigt uit om iedereen zo vaak mogelijk te mailen. Op korte termijn groeit met elk extra mailtje immers de kans op een verkoop.

Tip 2: Plaats de analisten in de business

Steeds weer zetten retailers het team van analisten bij elkaar in een kamer, soms zelfs als onderdeel

van de IT-afdeling. De afstand tot de mensen uit de business die de inzichten in praktijk moeten brengen, is groot. En te vaak blijkt die afstand onoverbrugbaar. Dat leidt tot misverstanden, onbegrepen analisten en waardevolle inzichten die onbenut blijven.

Beter is om de analisten samen met de mensen uit de business bij elkaar te zetten in multidisciplinaire teams, die werken met scrum-achtige technieken. Organisaties die succesvol zijn, beseffen dat ze continu in verandering moeten zijn en werken in dat soort teams. Dat betekent dat business managers in een vroegtijdig stadium worden betrokken bij de bouw van datamodellen, zodat analisten en de business van elkaar kunnen leren. Klantkennis zit immers in data én in mensen.

Tip 3: Neem een business analist in dienst

Data-analisten halen hun werkplezier vooral uit het maken van fraaie analyses en het opstellen van goede, misschien zelfs overontwikkelde datamodellen. Voor hun voldoening is het vaak niet eens nodig om de inzichten uit die modellen in praktijk te brengen. Veel analisten zijn daarom ook niet goed in het interpreteren van data en het vertalen daarvan naar de concrete impact op de retailer. 

Het kan verstandig zijn om daarom een business analist in te zetten. Dat is iemand die voldoende affiniteit heeft met analytics en enigszins snapt hoe datamodellen tot stand komen, maar ook weet wat de uitdagingen van de business managers zijn. Hij kan de kloof tussen analytics en business overbruggen door vragen uit de business te concretiseren en door inzichten uit datamodellen te vertalen naar kansen voor de retailer.

Tip 4: Analytics is een proces, geen project

Nog te veel retailers kijken naar alle inspanningen op het gebied van data en analytics alsof het een project met een kop en een staart betreft. Een project waarvan vooraf duidelijk moet zijn wat het gaat opleveren. Dat is vooral het geval bij retailorganisaties die worden geleid door managers uit de ‘oude generatie’ die onvoldoende gevoel en affiniteit met de nieuwe wereld hebben Het commitment van deze managers neemt snel af als investeringen in data en analytics niet snel genoeg resultaat opleveren.

Analytics is echter geen project, maar een proces waarin retailers met vallen en opstaan steeds handiger en slimmer worden. Een proces waarvan de uitkomst vooraf onduidelijk is, maar dat wel moet worden opgestart om vooruit te komen. Want alle ontwikkelingen in de retailmarkt maken één ding duidelijk: stilstand is achteruitgang.

Auteur: EY, Simon van Ulden, 5 oktober 2016

5 Astonishing IoT examples in civil engineering

The internet of things is making a major impact on the field of civil engineering, and these five examples of IoT applications in civil engineering are fascinating.

As the Internet of Things (IoT) becomes smarter and more advanced, we’ve started to see its usage grow across various industries. From retail and commerce to manufacturing, the technology continues to do some pretty amazing things in nearly every sector. The civil engineering field is no exception.

An estimated 20 billion internet-connected devices will be active around the world by 2020. Adoption is certainly ramping up, and the technologies that support IoT are also growing more sophisticated: including big data, cloud computing and machine learning.

As a whole, civil engineering projects have a lot to gain from the integration of IoT technologies and devices. The technology significantly improves automation and remote monitoring for many tasks, allowing operators to remain hands-off more than ever before. The data that IoT devices collect can inform and enable action throughout the scope of a project and even beyond.

For example, IoT sensors can monitor soil consolidation and degradation, as well as a development project’s environmental impact. Alternatively, IoT can measure and identify public roadways that need servicing. These two basic examples provide a glimpse into what IoT can do in the civil engineering sector.

IoT, alongside many other innovative construction technologies, will completely transform the industry. That said, what role is it currently playing in the field? What are some other applications that are either planned or now in use? How can the civil engineering industry genuinely use IoT?

1. Allows a transformation from reactionary to preventative maintenance

Most maintenance programs are corrective or reactionary. When something breaks down or fails, a team acts to fix the problem. In reality, this practice is nothing more than slapping a bandage on a gaping wound.

With development projects, once things start to break down, they generally continue on that path. Problems grow much more prominent, no matter what fixes you apply. It makes more sense, then, to monitor a subject’s performance and status and apply fixes long before things break down. In other words, using a preventative maintenance routine is much more practical, efficient and reliable.

IoT devices and sensors deliver all the necessary data to make such a process possible. They collect information about a subject in real-time and then report it to an external system or analytics program. That program then identifies potential errors and communicates the necessary information to a maintenance crew.

In any field of construction, preventative maintenance considerably improves the project in question as well as the entire management process. Maintenance management typically comprises about 40% to 50% of a business’s operational budget. Companies spend much of their time reacting to maintenance issues rather than preventing them. IoT can turn that around.

2. Presents a real-time construction management solution

A proper construction management strategy is necessary for any civil engineering project. Many nuanced tasks need to be completed, whether they involve tracking and measuring building supplies or tagging field equipment and dividing it up properly.

IoT technology can reduce tension by collecting relevant information in real time and delivering it to the necessary parties. Real-time solutions also provide faster time-to-action. Management and decision-makers can see almost immediately how situations are playing out and take action to either improve or correct a project’s course.

For example, imagine the following scenario. During a project that’s underway, workers hit a snag that forced them to use more supplies than expected. Rather than waiting until supplies run out, the technology has already ordered more. That way, the supplies are already on their way and will arrive at the project site before the existing supply is exhausted. The result is a seamless operation that continually moves forward, despite any potential errors. IoT can measure the number of supplies and report it to a remote system, which then makes the necessary purchase order.

3. Creates automated and reliable documentation

One of the minor responsibilities of development and civil engineering projects is related to paperwork. Documentation records a great deal about a project before, during and after it wraps up.

IoT technologies can improve the entire process, if not completely automate many of the tedious elements. Reports are especially useful to have during inspections, insurance and liability events, and much more. The data that IoT collects can be parsed and added to any report to fill out much-needed details. Because the process happens automatically, the reports can generate with little to no external input.

4. Provides a seamless project safety platform

Worksites can be dangerous, which is why supervisors and project managers must remain informed about their workers at all times. If an accident occurs, they must be able to locate and evacuate any nearby personnel. IoT can provide real-time tracking for all workers on a site, and even those off-site.

More importantly, IoT technology can connect all those disparate parties, allowing for direct communication with near-instant delivery. The result is a much safer operation for all involved, especially the workers who spend most of their time in the trenches.

5. Enhances operational intelligence support

By putting IoT and data collection devices in place with no clear guidance, an operation can suffer from data overload: an overabundance and complete saturation of intelligence with no clear way to analyze the data and use it.

Instead, once IoT technology is implemented, organizations are forced to focus on an improved operational intelligence program to make sure the data coming in is adequately vetted, categorized and put to use. It’s cyclical because IoT empowers the intelligence program by offering real-time collection and analysis opportunities. So, even though more data is coming in and the process of extracting insights is more complex, the reaction times are much faster and more accurate as a result.

Here’s a quick example. With bridge and tunnel construction, it’s necessary to monitor the surrounding area for environmental changes. Soil and ground movement, earthquakes, changes in water levels and similar events can impact the project. Sensors embedded within the surrounding area can collect pertinent information, which passes to a remote analytics tool. During a seismic event, the entire system would instantly discern if work must be postponed or if it can continue safely. A support program can distribute alerts to all necessary parties automatically, helping to ensure everyone knows the current status of the project, especially those in the field.

Identifying new opportunities with IoT

Most civil engineering and development teams have no shortage of projects in today’s landscape. Yet, it’s still crucial to remain informed about the goings-on to help pinpoint more practical opportunities.

When IoT is installed during new projects, the resulting data reports may reveal additional challenges or problems that would have otherwise gone unnoticed. A new two-lane road, for instance, may see more traffic and congestion than initially expected. Or, perhaps a recently developed water pipeline is seeing unpredictable pressure spikes.

With the correct solutions in place, IoT can introduce many new opportunities that might significantly improve the value and practicality of a project.

Author: Megan Ray Nichols

Source: SmartDataCollective

7 Personality assets required to be successful in data and tech

If you look at many of the best-known visionaries, such as Richard Branson, Elon Musk, and Steve Jobs, there are certain traits that they all have which are necessary for being successful. So this got me thinking, what are the characteristics necessary for success in the tech industry? In this blog, I’m going to explain the seven personality traits that I decided are necessary for success, starting with:

1. Analytical capabilities

Technology is extremely complex. If you want to be successful, you should be able to cope with complexity. Complexity not only from technical questions, but also when it comes to applying technology in an efficient and productive way.

2. Educational foundation

Part of the point above is your educational foundation. I am not talking so much about specific technical expertise learned at school or university, but more the general basis for understanding certain theories and relations. The ability to learn and process new information very quickly is also important. We all know that we have to learn new things continuously.

3. Passion

One of the most important things in the secret sauce for success is being passionate about what you do. Passion is the key driver of human activity, and if you love what you’re doing, you’ll be able to move mountains and conquer the world. If you are not passionate about what you are doing, you are doing the wrong thing.

4. Creativity

People often believe that if you are just analytical and smart, you’ll automatically find a good solution. But in the world of technology, there is no one single, optimal, rational solution in most cases. Creating technology is a type of art, where you have to look for creative solutions, rather than having a genius idea. History teaches us that the best inventions are born out of creativity.

5. Curiosity

The best technology leaders never stop being curious like children. Preserving an open mind, challenging everything and keeping your curiosity for new stuff will facilitate your personal success in a constantly changing world.

6. Persistence

If you are passionate, smart and creative and find yourself digging deeply into a technological problem, then you’ll definitively need persistence. Keep being persistent to analyze your problem appropriately, to find your solution, and eventually to convince others to use it.

7. Being a networker and team player

If you have all the other skills, you might already be successful. But, the most important booster of your success is your personal skillset. Being a good networker and team player, and having the right people in your network to turn to for support, will make the whole journey factors easier. There might be successful mavericks, but the most successful people in technology have a great set of soft skills.

As you’ll notice, these characteristics aren’t traits that you are necessarily born with. For those who find that these characteristics don’t come naturally to them, you’ll be pleased to hear that all can be learned and adopted through hard work and practice. Anyone can be successful in tech, and by keeping these traits in mind in future, you too can ensure a long and successful career in tech.

Author: Mathias Golombek

Source: Dataversity

9 Data issues to deal with in order to optimize AI projects

The quality of your data affects how well your AI and machine learning models will operate. Getting ahead of these nine data issues will poise organizations for successful AI models.

At the core of modern AI projects are machine-learning-based systems which depend on data to derive their predictive power. Because of this, all artificial intelligence projects are dependent on high data quality.

However, obtaining and maintaining high quality data is not always easy. There are numerous data quality issues that threaten to derail your AI and machine learning projects. In particular, these nine data quality issues need to be considered and prevented before issues arise.

1. Inaccurate, incomplete and improperly labeled data

Inaccurate, incomplete or improperly labeled data is typically the cause of AI project failure. These data issues can range from bad data at the source to data that has not been cleaned or prepared properly. Data might be in the incorrect fields or have the wrong labels applied.

Data cleanliness is such an issue that an entire industry of data preparation has emerged to address it. While it might seem an easy task to clean gigabytes of data, imagine having petabytes or zettabytes of data to clean. Traditional approaches simply don't scale, which has resulted in new AI-powered tools to help spot and clean data issues.

2. Having too much data

Since data is important to AI projects, it's a common thought that the more data you have, the better. However, when using machine learning sometimes throwing too much data at a model doesn't actually help. Therefore, a counterintuitive issue around data quality is actually having too much data.

While it might seem like too much data can never be a bad thing, more often than not, a good portion of the data is not usable or relevant. Having to go through to separate useful data from this large data set wastes organizational resources. In addition, all that extra data might result in data "noise" that can result in machine learning systems learning from the nuances and variances in the data rather than the more significant overall trend.

3. Having too little data

On the flip side, having too little data presents its own problems. While training a model on a small data set may produce acceptable results in a test environment, bringing this model from proof of concept or pilot stage into production typically requires more data. In general, small data sets can produce results that have low complexity, are biased or too overfitted and will not be accurate when working with new data.

4. Biased data

In addition to incorrect data, another issue is that the data might be biased. The data might be selected from larger data sets in ways that doesn't appropriately convey the message of the wider data set. In other ways, data might be derived from older information that might have been the result of human bias. Or perhaps there are some issues with the way that data is collected or generated that results in a final biased outcome.

5. Unbalanced data

While everyone wants to try to minimize or eliminate bias from their data sets, this is much easier said than done. There are several factors that can come into play when addressing biased data. One factor can be unbalanced data. Unbalanced data sets can significantly hinder the performance of machine learning models. Unbalanced data has an overrepresentation of data from one community or group while unnecessarily reducing the representation of another group.

An example of an unbalanced data set can be found in some approaches to fraud detection. In general, most transactions are not fraudulent, which means that only a very small portion of your data set will be fraudulent transactions. Since a model trained on this fraudulent data can receive significantly more examples from one class versus another, the results will be biased towards the class with more examples. That's why it's essential to conduct thorough exploratory data analysis to discover such issues early and consider solutions that can help balance data sets.

6. Data silos

Related to the issue of unbalanced data is the issue of data silos. A data silo is where only a certain group or limited number of individuals at an organization have access to a data set. Data silos can result from several factors, including technical challenges or restrictions in integrating data sets as well as issues with proprietary or security access control of data.

They are also the product of structural breakdowns at organizations where only certain groups have access to certain data as well as cultural issues where lack of collaboration between departments prevents data sharing. Regardless of the reason, data silos can limit the ability of those at a company working on artificial intelligence projects to gain access to comprehensive data sets, possibly lowering quality results.

7. Inconsistent data

Not all data is created the same. Just because you're collecting information, that doesn't mean that it can or should always be used. Related to the collection of too much data is the challenge of collecting irrelevant data to be used for training. Training the model on clean, but irrelevant data results in the same issues as training systems on poor quality data.

In conjunction with the concept of data irrelevancy is inconsistent data. In many circumstances, the same records might exist multiple times in different data sets but with different values, resulting in inconsistencies. Duplicate data is one of the biggest problems for data-driven businesses. When dealing with multiple data sources, inconsistency is a big indicator of a data quality problem.

8. Data sparsity

Another issue is data sparsity. Data sparsity is when there is missing data or when there is an insufficient quantity of specific expected values in a data set. Data sparsity can change the performance of machine learning algorithms and their ability to calculate accurate predictions. If data sparsity is not identified, it can result in models being trained on noisy or insufficient data, reducing the effectiveness or accuracy of results.

9. Data labeling issues

Supervised machine learning models, one of the fundamental types of machine learning, require data to be labeled with correct metadata for machines to be able to derive insights. Data labeling is a hard task, often requiring human resources to put metadata on a wide range of data types. This can be both complex and expensive. One of the biggest data quality issues currently challenging in-house AI projects is the lack of proper labeling of machine learning training data. Accurately labeled data ensures that machine learning systems establish reliable models for pattern recognition, forming the foundations of every AI project. Good quality labeled data is paramount to accurately training the AI system on what data it is being fed.

Organizations looking to implement successful AI projects need to pay attention to the quality of their data. While reasons for data quality issues are many, a common theme that companies need to remember is that in order to have data in the best condition possible, proper management is key. It's important to keep a watchful eye on the data that is being collected, run regular checks on this data, keep the data as accurate as possible, and get the data in the right format before having machine learning models learn on this data. If companies are able to stay on top of their data, quality issues are less likely to arise.

Author: Kathleen Walch

Source: TechTarget

9 Tips to become a better data scientist

Over the years I worked on many Data Science projects. I remember how easy it was to get lost and waste a lot of energy in the wrong direction. In time, I learned what works for me to be more effective. This list is my best try to sum it up:

1. Build a working pipeline first

While it’s tempting to start with the cool stuff first, you want to make sure that you don't spend time on small technical things like loading the data, feature extraction and so on. I like to start with a very basic pipeline, but one that works, i.e., I can run it end to end and get results. Later I expand every part while keeping the pipeline working.

2. Start simple and complicate one thing at a time

Once you have a working pipeline, start expanding and improving it. You have to take it to step by step. It is very important to understand what caused what. If you introduce too many changes at once, it will be hard to tell how each change affected the whole model. Keep the updates as simple and clean as possible. Not only it will be easier to understand its effect, but also, it will be easier to refactor it once you come up with another idea.

3. Question everything

Now you have a lot on your hands, you have a working pipeline and you already did some changes that improved your results. It’s important to understand why. If you added a new feature and it helped the model to generalize better, why? If it didn't, why not? Maybe your model is slower than before, why’s that? Are you sure each of your features/modules does what you think it does? If not, what happened?

These kinds of questions should pop in your head while you’re working. To end up with a really great result, you must understand everything that happens in your model.

4. Experience a lot and experience fast

After you questioned everything, you got stuck with… well, a lot of questions. The best way to answer them is to experiment. If you followed this far, you already have a working pipeline and a nicely written code, so conducting an experiment shouldn't waste much of your time. Ideally, you’ll be able to run more than one experiment at a time, this you’ll help you answer your questions and improve your intuition of what works and what is not.

Things to experiment: Adding/removing features, changing hyperparameters, changing architectures, adding/removing data and so on.

5. Prioritize and Focus

At this point, you did a lot of work, you have a lot of questions, some answers, some other tasks and probably some new ideas to improve your model (or even working on something entirely different).

But not all these are equally important. You have to understand what is the most beneficial direction for you. Maybe you came up with a brilliant idea that slightly improved your model but also made it much more complicated and slow, should you continue with this direction? It depends on your goal. If your goal is to publish a state-of-the-art solution, maybe it is. But if your goal is to deploy a fast and descent model to production, then probably you can invest your time on something else. Remember your final goal when working, and try to understand what tasks/experiments will get you closer to it.

6. Believe in your metrics

As discussed, understanding what’s working and what is not is very important. But how do you know when something works? you evaluate your results against some validation/test data and get some metric! You have to belive that metric! There may be some reasons not to believe in your metric. It could be the wrong one, for example. Your data may be unbalanced so accuracy can be the wrong metric for you. Your final solution must be very precise, so maybe you’re more interested in precision than recall. Your metric must reflect the goal you’re trying to achieve. Another reason not to believe in your metric is when your test data is dirty or noisy. Maybe you got data somewhere from the web and you don’t know exactly what’s in there?

A reliable metric is important to advance fast, but also, it’s important that the metric reflects your goals. In data science, it may be easy to convince ourselves that our model is good, while in reality, it does very little.

7. Work to publish/deploy

Feedback is an essential part of any work, and data science is not an exception. When you work knowing that your code will be reviewed by someone else, you’ll write much better code. When you work knowing that you’ll need to explain it to someone else, you’ll understand it much better. It doesn't have to be a fancy journal or conference or company production code. If you’re working on a personal project, make it open source, write a post about it, send it to your friends, show it to the world!

Not all feedback will be positive, but you’ll be able to learn from it and improve over time.

8. Read a lot and keep updated

I’m probably not the first one suggesting keeping up with the recent advancement to be effective, so instead of talking about it, I’ll just tell you how I do it: good old newsletters! I find them very useful as it’s essentially someone that keeps up with the most recent literature, picks the best stuff and sends it to you!

9. Be curious

While reading about the newest and coolest, don't limit yourself to the one area you’re interested in, try to explore others (but related) as well. It could be beneficial in a few ways. You can find a technique that works in one domain to be very useful in yours, you’ll improve your ability to understand complex ideas, and, you may find another domain that interests you so you’ll be able to expand your data skills and knowledge.

Conclusion

You’ll get much better results and enjoy the process if you’re effective. While all of the topics above are important, if I have to choose one, it will be 'Prioritize and Focus'. For me, all other topics lead to this one eventually. The key to success is to work on the right thing.

Author: Dima Shulga

Source: towards data science

A Closer Look at Generative AI

Artificial intelligence is already designing microchips and sending us spam, so what's next? Here's how generative AI really works and what to expect now that it's here.

Generative AI is an umbrella term for any kind of automated process that uses algorithms to produce, manipulate, or synthesize data, often in the form of images or human-readable text. It's called generative because the AI creates something that didn't previously exist. That's what makes it different from discriminative AI, which draws distinctions between different kinds of input. To say it differently, discriminative AI tries to answer a question like "Is this image a drawing of a rabbit or a lion?" whereas generative AI responds to prompts like "Draw me a picture of a lion and a rabbit sitting next to each other."

This article introduces you to generative AI and its uses with popular models like ChatGPT and DALL-E. We'll also consider the limitations of the technology, including why "too many fingers" has become a dead giveaway for artificially generated art.

The emergence of generative AI

Generative AI has been around for years, arguably since ELIZA, a chatbot that simulates talking to a therapist, was developed at MIT in 1966. But years of work on AI and machine learning have recently come to fruition with the release of new generative AI systems. You've almost certainly heard about ChatGPT, a text-based AI chatbot that produces remarkably human-like prose. DALL-E and Stable Diffusion have also drawn attention for their ability to create vibrant and realistic images based on text prompts. We often refer to these systems and others like them as models because they represent an attempt to simulate or model some aspect of the real world based on a subset (sometimes a very large one) of information about it.

Output from these systems is so uncanny that it has many people asking philosophical questions about the nature of consciousness—and worrying about the economic impact of generative AI on human jobs. But while all these artificial intelligence creations are undeniably big news, there is arguably less going on beneath the surface than some may assume. We'll get to some of those big-picture questions in a moment. First, let's look at what's going on under the hood of models like ChatGPT and DALL-E.

How does generative AI work?

Generative AI uses machine learning to process a huge amount of visual or textual data, much of which is scraped from the internet, and then determine what things are most likely to appear near other things. Much of the programming work of generative AI goes into creating algorithms that can distinguish the "things" of interest to the AI's creators—words and sentences in the case of chatbots like ChatGPT, or visual elements for DALL-E. But fundamentally, generative AI creates its output by assessing an enormous corpus of data on which it’s been trained, then responding to prompts with something that falls within the realm of probability as determined by that corpus.

Autocomplete—when your cell phone or Gmail suggests what the remainder of the word or sentence you're typing might be—is a low-level form of generative AI. Models like ChatGPT and DALL-E just take the idea to significantly more advanced heights.

Training generative AI models

The process by which models are developed to accommodate all this data is called training. A couple of underlying techniques are at play here for different types of models. ChatGPT uses what's called a transformer (that's what the T stands for). A transformer derives meaning from long sequences of text to understand how different words or semantic components might be related to one another, then determine how likely they are to occur in proximity to one another. These transformers are run unsupervised on a vast corpus of natural language text in a process called pretraining (that's the P in ChatGPT), before being fine-tuned by human beings interacting with the model.

Another technique used to train models is what's known as a generative adversarial network, or GAN. In this technique, you have two algorithms competing against one another. One is generating text or images based on probabilities derived from a big data set; the other is a discriminative AI, which has been trained by humans to assess whether that output is real or AI-generated. The generative AI repeatedly tries to "trick" the discriminative AI, automatically adapting to favor outcomes that are successful. Once the generative AI consistently "wins" this competition, the discriminative AI gets fine-tuned by humans and the process begins anew.

One of the most important things to keep in mind here is that, while there is human intervention in the training process, most of the learning and adapting happens automatically. So many iterations are required to get the models to the point where they produce interesting results that automation is essential. The process is quite computationally intensive. 

Is generative AI sentient?

The mathematics and coding that go into creating and training generative AI models are quite complex, and well beyond the scope of this article. But if you interact with the models that are the end result of this process, the experience can be decidedly uncanny. You can get DALL-E to produce things that look like real works of art. You can have conversations with ChatGPT that feel like a conversation with another human. Have researchers truly created a thinking machine?

Chris Phipps, a former IBM natural language processing lead who worked on Watson AI products, says no. He describes ChatGPT as a "very good prediction machine." Phipps says: 'It’s very good at predicting what humans will find coherent. It’s not always coherent (it mostly is) but that’s not because ChatGPT "understands." It’s the opposite: humans who consume the output are really good at making any implicit assumption we need in order to make the output make sense.'

Phipps, who's also a comedy performer, draws a comparison to a common improv game called Mind Meld: 'Two people each think of a word, then say it aloud simultaneously—you might say "boot" and I say "tree." We came up with those words completely independently and at first, they had nothing to do with each other. The next two participants take those two words and try to come up with something they have in common and say that aloud at the same time. The game continues until two participants say the same word.

Maybe two people both say "lumberjack." It seems like magic, but really it’s that we use our human brains to reason about the input ("boot" and "tree") and find a connection. We do the work of understanding, not the machine. There’s a lot more of that going on with ChatGPT and DALL-E than people are admitting. ChatGPT can write a story, but we humans do a lot of work to make it make sense.'

Testing the limits of computer intelligence

Certain prompts that we can give to these AI models will make Phipps' point fairly evident. For instance, consider the riddle "What weighs more, a pound of lead or a pound of feathers?" The answer, of course, is that they weigh the same (one pound), even though our instinct or common sense might tell us that the feathers are lighter.

ChatGPT will answer this riddle correctly, and you might assume it does so because it is a coldly logical computer that doesn't have any "common sense" to trip it up. But that's not what's going on under the hood. ChatGPT isn't logically reasoning out the answer; it's just generating output based on its predictions of what should follow a question about a pound of feathers and a pound of lead. Since its training set includes a bunch of text explaining the riddle, it assembles a version of that correct answer. But if you ask ChatGPT whether two pounds of feathers are heavier than a pound of lead, it will confidently tell you they weigh the same amount, because that's still the most likely output to a prompt about feathers and lead, based on its training set. It can be fun to tell the AI that it's wrong and watch it flounder in response; I got it to apologize to me for its mistake and then suggest that two pounds of feathers weigh four times as much as a pound of lead.

Why does AI art have too many fingers?

A notable quirk of AI art is that it often represents people with profoundly weird hands. The "weird hands quirk" is becoming a common indicator that the art was artificially generated. This oddity offers more insight into how generative AI does (and doesn't) work. Start with the corpus that DALL-E and similar visual generative AI tools are pulling from: pictures of people usually provide a good look at their face but their hands are often partially obscured or shown at odd angles, so you can't see all the fingers at once. Add to that the fact that hands are structurally complex—they're notoriously difficult for people, even trained artists, to draw. And one thing that DALL-E isn't doing is assembling an elaborate 3D model of hands based on the various 2D depictions in its training set. That's not how it works. DALL-E doesn't even necessarily know that "hands" is a coherent category of thing to be reasoned about. All it can do is try to predict, based on the images it has, what a similar image might look like. Despite huge amounts of training data, those predictions often fall short.

Phipps speculates that one factor is a lack of negative input: 'It mostly trains on positive examples, as far as I know. They didn't give it a picture of a seven fingered hand and tell it "NO! Bad example of a hand. Don’t do this." So it predicts the space of the possible, not the space of the impossible. Basically, it was never told to not create a seven fingered hand.'

There's also the factor that these models don't think of the drawings they're making as a coherent whole; rather, they assemble a series of components that are likely to be in proximity to one another, as shown by the training data. DALL-E may not know that a hand is supposed to have five fingers, but it does know that a finger is likely to be immediately adjacent to another finger. So, sometimes, it just keeps adding fingers. (You can get the same results with teeth.) In fact, even this description of DALL-E's process is probably anthropomorphizing it too much; as Phipps says, "I doubt it has even the understanding of a finger. More likely, it is predicting pixel color, and finger-colored pixels tend to be next to other finger-colored pixels."

Potential negative impacts of generative AI

These examples show you one of the major limitations of generative AI: what those in the industry call hallucinations, which is a perhaps misleading term for output that is, by the standards of humans who use it, false or incorrect. All computer systems occasionally produce mistakes, of course, but these errors are particularly problematic because end users are unlikely to spot them easily: If you are asking a production AI chatbot a question, you generally won't know the answer yourself. You are also more likely to accept an answer delivered in the confident, fully idiomatic prose that ChatGPT and other models like it produce, even if the information is incorrect. 

Even if a generative AI could produce output that's hallucination-free, there are various potential negative impacts:

• Cheap and easy content creation: Hopefully it's clear by now that ChatGPT and other generative AIs are not real minds capable of creative output or insight. But the truth is that not everything that's written or drawn needs to be particularly creative. Many research papers at the high school or college undergraduate level only aim to synthesize publicly available data, which makes them a perfect target for generative AI. And the fact that synthetic prose or art can now be produced automatically, at a superhuman scale, may have weird or unforeseen results. Spam artists are already using ChatGPT to write phishing emails, for instance.
• Intellectual property: Who owns an AI-generated image or text? If a copyrighted work forms part of an AI's training set, is the AI "plagiarizing" that work when it generates synthetic data, even if it doesn't copy it word for word? These are thorny, untested legal questions.
• Bias: The content produced by generative AI is entirely determined by the underlying data on which it's trained. Because that data is produced by humans with all their flaws and biases, the generated results can also be flawed and biased, especially if they operate without human guardrails. OpenAI, the company that created ChatGPT, put safeguards in the model before opening it to public use that prevent it from doing things like using racial slurs; however, others have claimed that these sorts of safety measures represent their own kind of bias.
• Power consumption: In addition to heady philosophical questions, generative AI raises some very practical issues: for one thing, training a generative AI model is hugely compute intensive. This can result in big cloud computing bills for companies trying to get into this space, and ultimately raises the question of whether the increased power consumption—and, ultimately, greenhouse gas emissions—is worth the final result. (We also see this question come up regarding cryptocurrencies and blockchain technology.)

Use cases for generative AI

Despite these potential problems, the promise of generative AI is hard to miss. ChatGPT's ability to extract useful information from huge data sets in response to natural language queries has search giants salivating. Microsoft is testing its own AI chatbot, dubbed "Sydney," though it's still in beta and the results have been decidedly mixed.

But Phipps thinks that more specialized types of search are a perfect fit for this technology. "One of my last customers at IBM was a large international shipping company that also had a billion-dollar supply chain consulting side business," he says.

Phipps adds: 'Their problem was that they couldn’t hire and train entry level supply chain consultants fast enough—they were losing out on business because they couldn’t get simple customer questions answered quickly. We built a chatbot to help entry level consultants search the company's extensive library of supply chain manuals and presentations that they could turn around to the customer.If I were to build a solution for that same customer today, just a year after I built the first one, I would 100% use ChatGPT and it would likely be far superior to the one I built. What’s nice about that use case is that there is still an expert human-in-the-loop double-checking the answer. That mitigates a lot of the ethical issues. There is a huge market for those kinds of intelligent search tools meant for experts.'

Other potential use cases include:

• Code generation: The idea that generative AI might write computer code for us has been bubbling around for years now. It turns out that large language models like ChatGPT can understand programming languages as well as natural spoken languages, and while generative AI probably isn't going to replace programmers in the immediate future, it can help increase their productivity.
• Cheap and easy content creation: As much as this one is a concern (listed above), it's also an opportunity. The same AI that writes spam emails can write legitimate marketing emails, and there's been an explosion of AI copywriting startups. Generative AI thrives when it comes to highly structured forms of prose that don't require much creativity, like resumes and cover letters.
• Engineering design: Visual art and natural language have gotten a lot of attention in the generative AI space because they're easy for ordinary people to grasp. But similar techniques are being used to design everything from microchips to new drugs—and will almost certainly enter the IT architecture design space soon enough.

Conclusion

Generative AI will surely disrupt some industries and will alter—or eliminate—many jobs. Articles like this one will continue to be written by human beings, however, at least for now. CNET recently tried putting generative AI to work writing articles but the effort foundered on a wave of hallucinations. If you're worried, you may want to get in on the hot new job of tomorrow: AI prompt engineering.

Author: Josh Fruhlinger

Source: InfoWorld 

A guide to Business Process Automation

Are you spending hours repeating the same tasks? Office workers spend 69 days a year on administrative tasks. You might be wishing for a simpler way to get those jobs done.

An increasing number of businesses are relying on automation tools to take those repetitive tasks off their plate. In fact, 67% of businesses said that software solutions would be important to remain competitive.

So, how will our workforce change with business process automation? And how will your business develop as the digital transformation era makes things happen faster?

In this complete guide, we’ll cover:

• What is Business Process Automation?
• 5 Business Process Automation examples
  • Accounting
  • Customer service
  • Employee onboarding
  • HR onboarding
  • Sales and marketing
• The benefits of Business Process Automation
• What business processes can be automated?
• Best practices with Business Process Automation

What is Business Process Automation?

Here’s a simple definition: Business Process Automation is the act of using software to make complex things simpler.

(It’s also known as BPA or BPM. The latter means Business Process Management.)

You can use BPA to cut the time you spend doing every-day tasks. For example, you can use chatbots to handle customer support queries. This uses robotic process automation (RPA). Or, you can use contract management software to get clients to put pen to paper on your deal.

How else can you use business process automation?

Business Process Automation examples

Accounting

Research has found that cloud computing reduces labor costs by 50%, which is probably why 67% of accountants prefer cloud accounting.

So, how can you use an accounting automation solution in your business?

• Generate purchase orders: Purchase orders have long paper trails that can be difficult to keep track of. Prevent that from becoming a problem by automating your purchase orders. Your software creates a PO and sends it automatically for approval.
• Handle accounts payable: Automating your 'accounts payable' department can take tedious payment-related jobs off your hands. The software scans your incoming invoices, records how much you need to pay, and pays it with the click of a button.
• Send invoices: Do you send the same invoices every week or month? Use automated invoicing systems to create business rules. For example, you can invoice your client on the 1st working day of each month without having to set a reminder to do it manually.

Customer service

Customer service is crucial for your business to get right. But it can take lots of human time, unless you’re taking advantage of these business process automations:

• E-mail and push notifications: Use machine learning software, like chatbots, to handle incoming messages. The technology will understand your customer inquiry, and respond within seconds. Your customers or business users don’t need to wait for a response from a human agent.
• Helpdesk support: Do you have an overwhelming log of support tickets? By automating your helpdesk, you can route tickets to different team members. For example, if someone says their query is about a billing issue, you could automatically send their ticket to a finance agent.
• Call center processes: Think about what tasksyour call center team. Chances are, they’ll send emails once they hang up the phone. Or, they’ll set reminders to contact their lead in a few days. You can automate those repetitive tasks for them to focus on money-making calls with new leads.

Employee onboarding

Lots of paperwork and decision-making is involved with bringing on a new team member. However, you can automate most of the onboarding process with automation software. Here are some use cases.

• Verify employment history: You don’t have to call a candidates’ references to verify they’ve worked there. You can automate this process using tools like PreCheck . This software scans data to find links between your candidates’ names and their past employers.
• Source candidates: Find the best candidates by automating your recruitment process. For example, you can post a job description to one profile and syndicate it to other listing websites.
• Manage contracts: Long gone are the days of posting an employment contract and waiting for your new team member to post it back. You can automate this business workflow with document signage software. It sends the document via email and automatically reminds your new team member to sign and return it. It simplifies the entire lifecycle of bringing a new team member on board.

(Some fear that automation will destroy jobs in this process. Forrester data goes against this: 10% of jobs will be lost, but 3% will be created.)

HR onboarding

Your Human Resources teamwork with people. But that doesn’t mean they have to manually do those people-related tasks themselves. You can use the HR process automation for things like:

• Time tracking: Figure out how much money you’re making per customer (or client) by tracking time. However, you can’t always rely on team members to record their time. It’s tricky to remember! You can automate their time-tracking, and use software to break down the time you’ve spent on each activity.
• Employee leave requests: Do your staff need to send an email to submit a PTO request? Those emails can get lost. Instead, use a leave management system. This software will accept or decline requests and manage shifts based on absences.
• Monitoring attendance: Keep an eye on your staff by using an automated attendance management system. You can track their clock-in (and out) times, breaks, and time off: without spying on them yourself.

Sales and marketing

Artificial Intelligence (AI) is the top growth area for sales teams, its’ adoption is expected to boost by 139% over the next 3 years. Your sales and marketing team can use business process automation for these sales and marketing activities:

• Lead nurturing: Don’t rely on sticky notes to remind you of the leads you’re nurturing. You can add them to a CRM. Then, use automation to follow-up with your leads using a premade template or social media message.
• Creating customer case studies: You can automate surveys to collect customer experience feedback. Add data processing software to pull sentiments from individual feedback submissions. From there, you can find customers likely to make the best case studies.
• A/B testing: You’re probably running A/B tests on your website to determine which elements work best. Automate that process using tools like Intellimize. They’ll automatically show variations to your visitors, and collect the real-time data to analyze. Pick the one with the best user experience metrics.

Still not convinced? This could give your business a competitive advantage. Just 28% of marketers use marketing automation software.

Benefits of Business Process Automation

The use cases we’ve shared work for any business. But they’re not just 'nice to have'. There are several ways you’ll benefit from business process automation, such as:

Increased efficiency and productivity: Your automation tools store information in the cloud. This means you can access your systems from anywhere. It’s great for remote or mobile workers who use multiple devices.

Faster turnaround: You don’t have to complete your day-to-day tasks manually. Sure, you’ll need to spend a few hours creating your automations. But you’ll save time when your software does them faster.

Cost savings: You might not think that the hours you spend doing certain tasks cost a lot in comparison to the software. But, those hours are salaried; you’re still paying each team member their hourly rate. McKinsey found that 45% of paid activities can be automated by technology. (That’s an equivalent of $2 trillion in total annual wages.)

Fewer errors: Some studies argue that computers are smarter than the human brain. In fact,  Google found that customers that use custom document classification have achieved up to 96% accuracy. You’re less prone to human errors using business process automation.

Better team collaboration: With automation software, your entire team can view the processes you’re making with their own account. They won’t need to wait for a suitable time to talk about strategy. They can check the automation processes to see for themselves. Again, this is great for distributed teams who don’t have in-office communication.

Best Practices with Business Process Automation

Ready to start using business automation software?

Avoid diving in feet-first with the first application you find. Refer to these best practices to get the most value out of process workflow automations:

• Know your business’ needs, and prioritize automation software that helps with them. For example, if your focus is improving customer wait times, look at chatbot-style automations.
• Write a list of the repetitive tasks, such as data entry, you’ll be able to automate. Do this by asking your team. the people who work in a specific department day-in, day-out. Or, ask your project management team for their advice. Can you find a single process or tool to streamline most of their tasks?
• Start training your entire team on how to use the process automations. Some applications offer this type of support as part of your purchase. IBM, for example, have a Skills Gateway.

The final thing to note? Don’t rush into business process automations.

Start small and get used to how software is used. Then, ask your team for feedback. It’s better to be safe than sorry with this type of business decision, especially when your business is at stake!

Author: Matt Shealy 

Source: SAP

Data centers have become a core component of modern living, by containing and distributing the information required to participate in everything from social life to economy. In 2017, data centers consumed 3 percent of the world’s electricity, and new technologies are only increasing their energy demand. The growth of high-performance computing — as well as answers to growing cyber-security threats and efficiency concerns — are dictating the development of the next generation of data centers.

But what will these new data centers need in order to overcome the challenges the industry faces? Here is a look at 5 major trends that will impact data center design in the future.

1. Hyperscale functionality

The largest companies in the world are increasingly consolidating computing power in massive, highly efficient hyperscale data centers that can keep up with the increasing demands of enterprise applications. These powerful data centers are mostly owned by tech giants like Amazon or Facebook, and there are currently around 490 of them in existence with more than 100 more in development. It’s estimated that these behemoths will contain more than 50 percent of all data that passes through data centers by 2021, as companies take advantage of their immense capabilities to implement modern business intelligence solutions and grapple with the computing requirements of the Internet of Things (IoT).

2. Liquid efficiency

The efficiency of data centers is both an environmental concern and a large-scale economic issue for operators. Enterprises in diverse industries from automotive design to financial forecasting are implementing and relying on machine-learning in their applications, which results in more expensive and high-temperature data center infrastructure. It’s widely known that power and cooling represent the biggest costs that data center owners have to contend with, but new technologies are emerging to combat this threat. Liquid cooling is swiftly becoming more popular for those building new data centers, because of its incredible efficiency and its ability to future-proof data centers against the increasing heat being generated by demand for high-performance computing. The market is expected to grow to $2.5 billion by 2025 as a result.

3. AI monitoring

Monitoring software that implements the critical advances made in machine learning and artificial intelligence is one of the most successful technologies that data center operators have put into practice to improve efficiency. Machines are much more capable of reading and predicting the needs of data centers second to second than their human counterparts, and with their assistance operators can manipulate cooling solutions and power usage in order to dramatically increase energy efficiency.

4. DNA storage

In the two-year span between 2015 and 2017, more data was created than in all of preceding history. As this exponential growth continues, we may soon see the sheer quantity of data outstrip the ability of hard drives to capture it. But researchers are exploring the possibility of storing this immense amount of data within DNA, as it is said that a single gram of DNA is capable of storing 215 million gigabytes of information. DNA storage could provide a viable solution to the limitations of encoding on silicon storage devices, and meet the requirements of an ever-increasing number of data centers despite land constraints near urban areas. But it comes with its own drawbacks. Although it has improved considerably, it is still expensive and extremely slow to write data to DNA. Furthermore, getting data back from DNA involves sequencing it, and decoding files and finding / retrieving specific files stored on DNA is a major challenge. However, according to Microsoft research data, algorithms currently being developed may lower the cost of sequencing and synthesizing DNA plunge to levels that make it feasible in the future.

5. Dynamic security

The average cost of a cyber-attack to the impacted businesses will be more than $150 million by 2020, and data centers are at the center of the modern data security fight. Colocation facilities have to contend with the security protocols of multiple customers, and the march of data into the cloud means that hackers can gain access to it through multiple devices or applications. New physical and cloud security features are going to be critical for the evolution of the data center industry, including biometric security measures on-site to prevent physical access by even the most committed thieves or hackers. More strict security guidelines for cloud applications and on-site data storage will be a major competitive advantage for the most effective data center operators going forward as cyber-attacks grow more costly and more frequent. The digital economy is growing more dense and complex every single day, and data center builders and operators need to upgrade and build with the rising demand for artificial intelligence and machine learning in mind. This will make it necessary for greener, more automated, more efficient and more secure data centers to be able to safely host the services of the next generation of digital companies.

Author: Gavin Flynn

Source: Information-management

From gene mapping to space exploration, humanity continues to generate ever-larger sets of data — far more information than people can actually process, manage, or understand.
Machine learning systems can help researchers deal with this ever-growing flood of information. Some of the most powerful of these analytical tools are based on a strange branch of geometry called topology, which deals with properties that stay the same even when something is bent and stretched every which way.

Such topological systems are especially useful for analyzing the connections in complex networks, such as the internal wiring of the brain, the U.S. power grid, or the global interconnections of the Internet. But even with the most powerful modern supercomputers, such problems remain daunting and impractical to solve. Now, a new approach that would use quantum computers to streamline these problems has been developed by researchers at MIT, the University of Waterloo, and the University of Southern California.
The team describes their theoretical proposal this week in the journal Nature Communications. Seth Lloyd, the paper’s lead author and the Nam P. Suh Professor of Mechanical Engineering, explains that algebraic topology is key to the new method. This approach, he says, helps to reduce the impact of the inevitable distortions that arise every time someone collects data about the real world.

In a topological description, basic features of the data (How many holes does it have? How are the different parts connected?) are considered the same no matter how much they are stretched, compressed, or distorted. Lloyd explains that it is often these fundamental topological attributes “that are important in trying to reconstruct the underlying patterns in the real world that the data are supposed to represent.”

It doesn’t matter what kind of dataset is being analyzed, he says. The topological approach to looking for connections and holes “works whether it’s an actual physical hole, or the data represents a logical argument and there’s a hole in the argument. This will find both kinds of holes.”
Using conventional computers, that approach is too demanding for all but the simplest situations. Topological analysis “represents a crucial way of getting at the significant features of the data, but it’s computationally very expensive,” Lloyd says. “This is where quantum mechanics kicks in.” The new quantum-based approach, he says, could exponentially speed up such calculations.

Lloyd offers an example to illustrate that potential speedup: If you have a dataset with 300 points, a conventional approach to analyzing all the topological features in that system would require “a computer the size of the universe,” he says. That is, it would take 2300 (two to the 300th power) processing units — approximately the number of all the particles in the universe. In other words, the problem is simply not solvable in that way.
“That’s where our algorithm kicks in,” he says. Solving the same problem with the new system, using a quantum computer, would require just 300 quantum bits — and a device this size may be achieved in the next few years, according to Lloyd.

“Our algorithm shows that you don’t need a big quantum computer to kick some serious topological butt,” he says.
There are many important kinds of huge datasets where the quantum-topological approach could be useful, Lloyd says, for example understanding interconnections in the brain. “By applying topological analysis to datasets gleaned by electroencephalography or functional MRI, you can reveal the complex connectivity and topology of the sequences of firing neurons that underlie our thought processes,” he says.

The same approach could be used for analyzing many other kinds of information. “You could apply it to the world’s economy, or to social networks, or almost any system that involves long-range transport of goods or information,” says Lloyd, who holds a joint appointment as a professor of physics. But the limits of classical computation have prevented such approaches from being applied before.

While this work is theoretical, “experimentalists have already contacted us about trying prototypes,” he says. “You could find the topology of simple structures on a very simple quantum computer. People are trying proof-of-concept experiments.”

Ignacio Cirac, a professor at the Max Planck Institute of Quantum Optics in Munich, Germany, who was not involved in this research, calls it “a very original idea, and I think that it has a great potential.” He adds “I guess that it has to be further developed and adapted to particular problems. In any case, I think that this is top-quality research.”
The team also included Silvano Garnerone of the University of Waterloo in Ontario, Canada, and Paolo Zanardi of the Center for Quantum Information Science and Technology at the University of Southern California. The work was supported by the Army Research Office, Air Force Office of Scientific Research, Defense Advanced Research Projects Agency, Multidisciplinary University Research Initiative of the Office of Naval Research, and the National Science Foundation.

Source:MIT news

Predictive analytics / machine learning / artificial intelligence is a hot topic – what’s it about?

Using algorithms to help make better decisions has been the “next big thing in analytics” for over 25 years. It has been used in key areas such as fraud the entire time. But it’s now become a full-throated mainstream business meme that features in every enterprise software keynote — although the industry is battling with what to call it.

It appears that terms like Data Mining, Predictive Analytics, and Advanced Analytics are considered too geeky or old for industry marketers and headline writers. The term Cognitive Computing seemed to be poised to win, but IBM’s strong association with the term may have backfired — journalists and analysts want to use language that is independent of any particular company. Currently, the growing consensus seems to be to use Machine Learning when talking about the technology and Artificial Intelligence when talking about the business uses.

Whatever we call it, it’s generally proposed in two different forms: either as an extension to existing platforms for data analysts; or as new embedded functionality in diverse business applications such as sales lead scoring, marketing optimization, sorting HR resumes, or financial invoice matching.

Why is it taking off now, and what’s changing?

Artificial intelligence is now taking off because there’s a lot more data available and affordable, powerful systems to crunch through it all. It’s also much easier to get access to powerful algorithm-based software in the form of open-source products or embedded as a service in enterprise platforms.

Organizations today have also more comfortable with manipulating business data, with a new generation of business analysts aspiring to become “citizen data scientists.” Enterprises can take their traditional analytics to the next level using these new tools.

However, we’re now at the “Peak of Inflated Expectations” for these technologies according to Gartner’s Hype Cycle — we will soon see articles pushing back on the more exaggerated claims. Over the next few years, we will find out the limitations of these technologies even as they start bringing real-world benefits.

What are the longer-term implications?

First, easier-to-use predictive analytics engines are blurring the gap between “everyday analytics” and the data science team. A “factory” approach to creating, deploying, and maintaining predictive models means data scientists can have greater impact. And sophisticated business users can now access some the power of these algorithms without having to become data scientists themselves.

Second, every business application will include some predictive functionality, automating any areas where there are “repeatable decisions.” It is hard to think of a business process that could not be improved in this way, with big implications in terms of both efficiency and white-collar employment.

Third, applications will use these algorithms on themselves to create “self-improving” platforms that get easier to use and more powerful over time (akin to how each new semi-autonomous-driving Tesla car can learn something new and pass it onto the rest of the fleet).

Fourth, over time, business processes, applications, and workflows may have to be rethought. If algorithms are available as a core part of business platforms, we can provide people with new paths through typical business questions such as “What’s happening now? What do I need to know? What do you recommend? What should I always do? What can I expect to happen? What can I avoid? What do I need to do right now?”

Fifth, implementing all the above will involve deep and worrying moral questions in terms of data privacy and allowing algorithms to make decisions that affect people and society. There will undoubtedly be many scandals and missteps before the right rules and practices are in place.

What first steps should companies be taking in this area?
As usual, the barriers to business benefit are more likely to be cultural than technical.

Above all, organizations need to make sure they have the right technical expertise to be able to navigate the confusion of new vendors offers, the right business knowledge to know where best to apply them, and the awareness that their technology choices may have unforeseen moral implications.

Source: timoelliot.com, October 24, 2016

A word of advice to help you get your first data science job

Creativity, grit, and perseverance will become the three words you live by

Whether you’re a new graduate, someone looking for a career change, or a cat similar to the one above, the data science field is full of jobs that tick nearly every box on the modern worker’s checklist. Working in data science gives you the opportunity to have job security, a high-paying salary with room for advancement, and the ability to work from anywhere in the world. Basically, working in data science is a no-brainer for those interested.

However, during the dreaded job search, many of us run into a situation where experience is required to be hired while in order to gain experience you need to be hired first...

Pretty familiar, right?

Having run into many situations myself where companies are often looking for candidates with 20 years of work experience before the age of 22, I understand the aggravation that comes with trying to look for a job when you’re a new graduate, someone looking for a career change, or even a cat, with no relevant work experience.

However, this is no reason to become discouraged. While many data science jobs require work experience, there are plenty of ways to create your own work experience that will make you an eligible candidate for these careers.

All you need is a little creativity, grit, and perseverance.

Adopting Data Science in a Business Environment

While most organizations understand the importance of data, far fewer have figured out how to successfully become a data-driven company. It’s enticing to focus on the “bells and whistles” of machine learning and artificial intelligence algorithms that can take raw data and create actionable insights. However, before you can take advantage of advanced analytics tools, there are other stops along the way, from operational reporting to intelligent learning.

Digital transformation is dependent on adoption. But adoption and proficiency of new technologies can be disruptive to an organization. Mapping a data journey provides awareness and understanding of where your organization is to ultimately get where you want to go, with enablement and adoption of the technology throughout. Without the clarity provided by a data journey, your organization won’t be positioned to successfully deploy the latest technology.

Here are the four elements of an effective data journey. 

Determine Your Roadmap

As with any trip, your data journey requires a roadmap to get you from where you are to where you want to go. Before you can get to your destination, the first step is to assess where you are.

Most organizations begin with a focus on operational reports and dashboards, which can help you glean business insights from what happened, including how many products were sold, how often and where. They can also identify where problems exist, and deliver alerts about what actions are needed.

Ultimately, most want to get to the point where analytics tools can help with statistical analysis, forecast, predictive analytics and optimization. Armed with machine learning, manufacturers want to understand why something is happening, what happens if trends continue, what’s going to happen next and what’s the best that can be done.

Capture Data and Build Processes and Procedures

Once you know where you want to go, it’s important to capture the data that is essential in helping you achieve your business goals. Manufacturers capture tremendous amounts of data, but if the data you collect doesn’t solve a business need, it’s not vital to your data processing priorities.

This phase of your data journey isn’t just about what data you collect, it’s also about your data strategy: how you collect the data, pre-process it, protect it and safely store it. You need to have processes and procedures in place to handle data assets efficiently and safely. Questions such as how you can leverage the cloud to gain access to data management tools, data quality and data infrastructure need to be answered. 

Make Data Accessible to Business Users

Today, data – and business insights about that data – need to be accessible to business users. This democratization of data makes it possible for business users from procurement to sales and marketing to access the data that’s imperative for them to do their jobs more effectively.

In the past, data was the domain of specialists which often caused bottlenecks in operations while they analyzed the data. In this phase of the data journey, it’s important to consider data management tools that can consolidate and automate data collection and analysis.

Once data is removed from silos, it makes it possible for data to be analyzed by more advanced analytics and data science tools to glean business insights that can propel your success. 

Change Company Culture for Full Adoption

A data culture gap is a common barrier to the adoption of advanced data analytics tools for many companies. When employees who are expected to use the data and insights don’t understand the benefits data can bring to decision-making it can create a roadblock. Your company won’t be data-driven until your team embraces a data-driven culture and starts to use the data intelligently.

If you want to get the most out of the advanced data analytics tools that are available today and use data intelligently in your organization, you must first develop a solid foundation.

First, you must be clear where you are in your organization’s data journey with a roadmap. Then create effective data processes, procedures, and collection methods, as well as identify what data management and analytics tools can support your initiatives. Finally, your team is key to adopting advanced data analytics tools, so be sure they are trained and understand how these tools can empower them. Once you have a solid analytics foundation, you’re ready to put machine learning to work to drive your collective success.

Author: Michael Simms

Source: Insidebigdata

AI in civil engineering: fundamentals, applications and developments

Artificial intelligence has always been a far-reaching manifold technology with limitless potential across industries. AI in civil engineering took a central stage a long time ago with the advent of complex constructions such as skyscrapers. Today, we are witnessing the wide-scale adoption of artificial intelligence in civil engineering, with smart algorithms, Big data, and deep learning techniques redefining productivity performance.

With that said, let’s dwell on the current application of AI in civil engineering as well as go over the basics of amplifying construction with machine intelligence.

Artificial intelligence in civil engineering: fundamentals

According to McKinsey, the construction sector is one of the largest in the world economy. Its spending amounts to around $10 trillion that goes for construction-related goods and services every year. This number doesn’t seem so monstrous, though. A lion’s share of this amount is justified by the growing and much-needed tech innovations.

AI software development for construction is no different from other verticals. It is an umbrella term associated with machines developing human-like functions. The latter may include everything from problem-solving to pattern recognition.

However, machine learning in civil engineering is what steals the show since it lays the ground for most smart techniques in construction.

Machine learning in civil engineering

Construction projects pose a unique set of challenges due to their scale and the number of contractors involved. That is why civil engineering companies are turning to machine learning and data science consulting to help with the construction and design of roads, bridges, and other infrastructure projects. Historically, some machine learning algorithms were more popular than others in the field.

Evolutionary computation (EC)

Evolutionary modeling or computation is an AI category based on principles and concepts of evolutionary biology (i.e. Darwinian) and population genetics. Thanks to an iterative process, it offers an effective way to tackle complex optimization problems. This machine learning technique is widely applied in design engineering to automate design production. The typical evolutionary models used in construction include Genetic Algorithms, Artificial Immune Systems, and Genetic Programming.

Artificial neural networks (ANNs)

ANNs exhibit excellent performance in lots of areas, including construction. Artificial neural networks are modeled after the brain and can be trained to recognize patterns. This makes them useful for tasks such as decision making, pattern recognition, forecasting, data analysis. Civil engineering includes all those tasks. Thus, ANNs are widely present in studying building materials, defect detection, geotechnical engineering, and construction management.

Fuzzy systems

A fuzzy system is a way of reasoning that mimics the human way of thinking. It helps machines deal with inexact input and output in construction projects. These algorithms allow companies to model the cost, time, and risk of construction. Thus, fuzzy systems are also used for quality assessment of infrastructure projects at conceptual cost estimating stages.

Moreover, fuzzy logic is applicable for:

• Finding performance deviations during construction and forecasting relevant corrective measures
• Analyzing the impact of construction delays and adjusting the schedule estimating design cost overruns
• Identifying mark-ups for competitive bids
• Improving industrial fabrication and modularisation procedures, etc.

Expert system

Expert systems are also one of the most popular machine learning techniques for civil engineering problems. As such, the algorithm is based on the existing knowledge corpus of human professional experts to establish a knowledge system. This technique is widely employed in construction engineering, underground and geotechnical engineering as well as geological exploration. Thus, these algorithms can analyze the energy consumption of a certain building or group of buildings and offer suggestions for energy sources.

Overall, thanks to the growing adoption of machine learning techniques for civil engineering problems, AI in the construction market is projected to be worth over $2312 million by 2026. Some typical areas where AI is being used include highway design, traffic management, and construction planning.

Now let’s get over the real-life examples of artificial intelligence in buildings construction to further illustrate the significance of this technology.

Top engineering applications of artificial intelligence

The potential applications of AI in civil engineering are vast and diverse. From optimizing processes and improving product design, to automating tasks and reducing waste, AI has the potential to make a significant impact within the sector. Here are the most promising applications of AI within engineering.

Smart construction design

Constructing a building isn’t a one-day task that involves lots of pre-planning. Sometimes, it may take years to bring a particular vision to life. Therefore, the planning stage in construction has a lot to benefit from smart systems combined with Big data technologies.

Thus, AI-enabled tools and programs can now automate the calculation and environmental analysis. Instead of manually compiling weather data, material properties, and others, architects can automatically pull necessary data. Parametric design, for instance, has been one of the fields that have benefited the most from automated workflows.

Moreover, artificial intelligence has strengthened the core 3D construction system called BIM. BIM or Building Information Modeling allows architects to create data-laden models based on the comprehensive information layer.

The latter helps automatically create drawings and reports, perform project analysis, simulate the schedule of works, operation of facilities, and others. Due to unmatched analytical and future-telling abilities, smart algorithms can also assess resource-efficient solutions and create low-risk execution plans.

Moreover, machine intelligence can take the form of virtual and augmented reality. Both are now finding adoption in architecture to walk clients through lifelike experiences with ready designs. This way, clients have a better vision of the future project and can give actional feedback on further improvements without spending extra money.

Construction process orchestration

On-site construction management has always been agonizing for construction firms. According to McKinsey, mismanagement of building processes costs the construction industry $1.6 trillion a year. So, after using machine intelligence to assess structural damages in Mexico City after the earthquake, engineers have readily employed algorithms in other aspects of construction.

Autonomous construction monitoring with robots and UAVs is what makes real-time remote monitoring of construction sites possible. Unmanned aerial vehicles or drones fly over the construction sites and map the area with high-resolution cameras. After that, the system generates a 3D map and a report to be shared via the cloud with stakeholders. Also, drone maps’ geo tagging capabilities allow for the acquisition of relevant area measurements and the conversion of those measurements into an estimated stockpile volume for decision-making.

This way, both decision-makers, and workers stay safe and have a holistic vision of the ongoing progress. At the moment, autonomous inspection is not adopted en-masse due to the tech and resource limitations. However, the COVID-19 pandemic has accelerated the adoption by bringing automated systems on-site to check workers for symptoms and epidemiological factors.

Smart cities

Away from on-site usage to fundamental application, smart cities can be heralded as one of the most exciting and bold engineering applications of artificial intelligence. A smart city is a man-made interconnected system of information and communication technologies. This tech biota is inhabited by IoT and artificial intelligence to facilitate the management of internal urban processes. Ultimately, the main goal of this system is to make our lives more comfortable and safe.

To make our cities smarter, we need to use all the tools at our disposal. One of these tools is artificial intelligence (AI), which is being used more and more in smart city construction projects. For example, by using AI, planners can better understand how people move around cities and what kind of services and facilities they need. This helps to optimize the design of smart cities, making them more efficient and user-friendly.

Yet, the implementation of artificial intelligence in the smart urban landscape is more present in ready infrastructure. However, smart systems are still necessary at the construction stage to lay the ground for smart cities and design them in a technology-friendly way.

Construction 3D printing

Last but not least is a mind-boggling application of machine intelligence in 3D printing. In architecture, the construction of a building is a huge and costly undertaking. Not only do the architects have to design the building, but the engineers also need to calculate how it will stand up to wind loads, seismic forces, and other environmental stresses. Moreover, builders must find ways to make these structures not just habitable, but comfortable to live in, with features like air conditioning and insulation.

With 3D printing, a large part of the construction process is automated. As such, 3D house printing is the process of printing a three-dimensional object using a 3D printer. The object is printed by laying down successive layers of material until the entire object is created. 3D printers are now being used for house construction. The use of 3D printers for house construction has many advantages, including reduced waste and lower costs.

Thus, the AI Build company has already developed an AI-based 3D printing technology that can print large 3D objects at high speed and with great accuracy.

What awaits artificial intelligence in the construction field?

As we see from the current application areas, the future of AI in civil engineering is shrouded in potential but fraught with uncertainty. There are a number of ways that AI could be deployed within the civil engineering field, from the design and analysis of structures to the monitoring and maintenance of infrastructure.

However, due to the limited adoption, it is still hard to assess the whole significance of the technology for construction. Moreover, the use of artificial intelligence in civil engineering is still in its early days. Nonetheless, smart algorithms have great potential to improve the safety and efficiency of civil engineering projects. As AI technology continues to evolve, the possibilities for its application in civil engineering will continue to grow.

Author: Tatsiana Isakova

Source: InData Labs

AI: The Game Changer in Online Businesses

AI technology is unquestionably changing the future of business. A growing number of companies are revamping their online business models to deal with new AI tools. Therefore, it should not be surprising to hear that the market for AI is expected to grow from $100 billion in 2021 to over $2 trillion in 2030.

Traditional company growth techniques depended mainly on face-to-face encounters, print marketing, and word-of-mouth recommendations before the Internet. Yet, the digital revolution has resulted in a radical shift, changing the corporate landscape in unimaginable ways. This article delves into how the Internet has impacted and transformed traditional business development strategies:

• Shift from Traditional Advertising to Digital Marketing
• The Decline of Print, Radio, and Television Advertising

Before the Internet, businesses relied heavily on print, radio, and television advertising to reach their target audience. Yet, due to the spread of digital platforms and the inevitable growth of social media, these traditional advertising methods have seen a cascading decrease in both reach and performance.

The rise of AI technology has only exacerbated this trend. Keep reading to learn more about the effect AI is having on Internet business models.

Rise of AI-Driven Social Media Marketing

AI has led to a number of improvements for social media marketers, especially when it comes to ad management and optimization. AI-powered tools can analyze numerous ad targeting and budget variations, segment audiences, create ad creative, test ads, and enhance speed and performance in real-time for optimal results.

Rem Darbinyan is the Founder and CEO of SmartClick, talked about some of the benefits of using AI in social media marketing in a post on Forbes. He points out that artificial intelligence is changing our lives, especially in social media. Social media platforms such as Facebook and Instagram use AI for content moderation, personalized recommendations, and ads. There are 4.26 billion active social media users worldwide. They spend an average of 2 hours and 27 minutes daily on social media platforms. As social media users grow, the need for AI solutions to understand customer preferences is also increasing. The AI market in social media is expected to reach $3,714.89 million by 2026, with a CAGR of 28.77%. While the social media platforms themselves use AI technology, businesses also leverage social media tools to get the most from social media.

The development of social media has created an opportunity for businesses to engage with their consumers in a more focused and engaging manner. The ROI is higher than ever, now that so many businesses can use AI tools like HootSuite and Buffer. This novel marketing approach encompasses several tactics that involve AI technology:

• Targeted Advertising: Social media platforms offer granular targeting options, enabling businesses to reach specific demographics based on age, location, and interests.
• Influencer Marketing: By collaborating with influential personalities with a substantial online following, businesses can promote their products and services more authentically, leveraging the trust and credibility these influencers have cultivated with their audience.
• User-Generated Content: Encouraging customers to create and share content related to a brand helps foster a sense of community and enhances the brand’s credibility, effectively turning customers into brand advocates.

AI technology is going to continue to impact the future of social media marketing for years to come.

Growing Importance of AI for Search Engine Optimization

As search engines have become the primary gateway for users to access information, businesses must ensure their websites rank prominently in search engine results to attract potential customers. SEO involves optimizing a website’s structure, content, and user experience to improve visibility and achieve higher organic search rankings. Moz provides a comprehensive guide to understanding the intricacies of SEO and implementing effective strategies.

AI is especially useful in SEO. A 2021 report by the American Marketing Association found that 80% of SEO professionals intend to use AI. Of course, AI technology really took off in 2022, so that figure has probably increased substantially in the past year.

AI can help improve your SEO strategy by discovering opportunities, such as helping businesses find relevant keywords to get more organic traffic to their platforms. AI SEO tools can expedite the process and enhance the accuracy of keyword research, competitor analysis, and search intent research.

AI can improve the accuracy, efficiency, and performance of SEO strategies, including content production. Many SEO professionals have been using tools like ChatGPT to scale content production considerably. AI serves as a supporting tool, not a replacement for SEOs. AI tools can be used to complete dozens of functions and analyze billions of data points making it a smart step for any SEO strategy.

Early adopters of AI for SEO can benefit the most by creating data-backed content that interests readers and aligns with search engine algorithms.

Role of AI in Content Marketing

Content marketing focuses on creating and distributing valuable, relevant, and consistent content to attract and retain a clearly defined audience, ultimately driving profitable customer action. By crafting informative and engaging content, businesses can establish themselves as thought leaders in their respective industries, building trust with their audience and fostering long-term customer relationships. Content Marketing Institute offers valuable insights into content marketing and its various facets.

AI technology can invaluable for many parts of the content marketing practice. As stated above, many businesses use tools like ChatGPT to scale content production. However, they can also use AI tools like Grammarly to improve the quality of their content.

E-commerce and the Transformation of Retail

AI Leads to Growth in Online Shopping

The Internet has radically transformed the retail landscape by facilitating the growth of online shopping. Customers may now buy items and services from the comfort of their own homes, with nearly limitless options and very competitive pricing. According to Statista, global e-commerce sales have been steadily increasing, highlighting the importance of having a solid online presence for businesses of all sizes.

Evolution of Brick-and-Mortar Stores

Despite the rising dominance of e-commerce, brick-and-mortar retailers continue to change and adapt to the changing landscape through a variety of strategies:

• Omnichannel Retail Strategies: This approach entails integrating various online and offline sales channels to provide customers with a seamless and consistent shopping experience. For example, retailers may offer services such as buying online, picking up in-store (BOPIS), or in-store returns for online purchases.
• Experiential Retail: To differentiate themselves from online retailers, brick-and-mortar stores increasingly focus on creating unique and immersive customer experiences. This can include hands-on product demonstrations, interactive displays, and in-store events.

Impact on Supply Chain Management

E-commerce development has also resulted in changes in supply chain management strategies, with enterprises adopting new models such as:

• Drop-shipping: In this model, retailers do not hold inventory but transfer customer orders and shipment details to manufacturers or wholesalers, who then ship the products directly to customers. This allows businesses to minimize inventory costs and mitigate the risks of holding stock.
• Just-in-Time Inventory Management: This strategy involves closely monitoring inventory levels and ordering products only when needed, reducing the amount of stock held on hand and minimizing storage costs.

Online Business Loans and Alternative Financing

The Rise of Online Lenders

Traditionally, businesses seeking financing would approach banks and other financial institutions for loans. However, the emergence of online lenders has revolutionized the borrowing landscape, offering faster application processes and more flexible loan options. Moreover, online lenders offer some unique advantages over traditional lenders, making them an increasingly popular choice for businesses in need of capital.

Advantages of Online Business Loans

• Access to Capital for Small Businesses: Online lenders often have less strict eligibility requirements than traditional banks, making it easier for small businesses to secure funding.
• Competitive Interest Rates: Due to lower overhead costs and increased competition, online lenders often offer competitive interest rates and more favorable loan terms.

Crowdfunding and Peer-to-Peer Lending

Crowdfunding and peer-to-peer (P2P) lending platforms provide alternative financing options for businesses by connecting them directly with investors or individuals willing to lend money. These platforms allow businesses to raise capital without relying on traditional financial institutions while also providing an avenue for investors to support innovative ideas and earn returns on their investments.

Invoice Financing and Other Innovative Solutions

Invoice financing allows businesses to receive immediate cash advances on outstanding invoices, helping to alleviate cash flow issues that often arise from delayed payments. Companies like Fundbox and BlueVine specialize in providing invoice financing services, enabling businesses to maintain their working capital and continue growing.

Remote Work and the Global Talent Pool

Advantages of Remote Work for Businesses

The widespread adoption of the Internet has facilitated the rise of remote work, bringing numerous benefits to businesses, such as:

• Lower Overhead Costs: By embracing remote work, companies can significantly reduce costs associated with office space, utilities, and other operational expenses.
• Access to a Wider Range of Talent: Remote work enables businesses to tap into a global talent pool, unshackled by geographical constraints. This allows them to find highly skilled professionals that may not be available in their immediate vicinity.

Impact on Company Culture and Communication

Remote work also necessitates a shift in company culture and communication practices. Businesses must foster an environment of trust and autonomy while implementing practical communication tools and strategies to ensure seamless collaboration among remote team members. Resources like Remote.co provide valuable insights and best practices for managing remote teams and maintaining a strong company culture.

Use of Collaborative Tools and Software

To maintain productivity and collaboration among remote teams, businesses must leverage a suite of digital tools and software. These may include project management tools like Asana or Trello, communication platforms like Slack or Microsoft Teams, and file-sharing services like Google Drive or Dropbox.

Leveraging Data Analytics for Business Development

Importance of Data-Driven Decision Making

Businesses in today’s hyper-connected world have access to massive volumes of data that may be used to make educated decisions and drive development. Data-driven decision-making is gathering, analyzing, and interpreting data in order to find patterns, trends, and insights that may be used to influence strategic business choices.

Customer Segmentation and Personalized Marketing

Businesses may segment their consumer base into various groups based on demographics, tastes, and behaviors by leveraging data analytics. This enables them to tailor their marketing efforts, creating personalized campaigns that resonate with their target audience and foster customer loyalty. Segment is a platform that helps businesses implement effective customer segmentation strategies.

Predictive Analytics for Forecasting Trends and Demand

Predictive analytics utilizes advanced data mining techniques machine learning, and statistical algorithms to forecast future trends, customer demand, and market conditions. This enables businesses to make proactive decisions, optimize their operations, and mitigate potential risks.IBM provides comprehensive solutions for businesses seeking to incorporate predictive analytics into their decision-making processes.

Cybersecurity and Data Privacy

Increased Risk of Cyber Threats for Businesses

As businesses increasingly rely on digital platforms and store sensitive data online, they become more susceptible to cyber threats, such as data breaches, ransomware attacks, and phishing scams. These cyber incidents can result in significant financial losses, reputational damage, and legal repercussions.

Importance of Data Protection and Privacy Regulations

To safeguard customer data and ensure compliance with data protection and privacy regulations, such as the General Data Protection Regulation (GDPR) and the California Consumer Privacy Act (CCPA), businesses must implement robust data security measures and adhere to industry best practices.

Best Practices for Maintaining a Secure Online Presence

To maintain a secure online presence and mitigate the risk of cyber threats, businesses should consider the following best practices:

• Implement strong authentication measures, such as multi-factor authentication and secure password policies.
• Regularly update software, applications, and operating systems to patch security vulnerabilities.
• Encrypt sensitive data during transmission and storage to protect it from unauthorized access.
• Conduct regular security audits and vulnerability assessments to identify and address potential weaknesses.
• Educate employees about cybersecurity best practices and the importance of maintaining a secure online environment.

Organizations like the National Institute of Standards and Technology (NIST) and the Center for Internet Security (CIS) offer valuable resources and guidelines for businesses looking to bolster their cybersecurity posture.

AI Advances Are Driving Changes in Online Business

The Internet has irrevocably changed the way businesses operate and develop their strategies. New advances in AI technology are accelerating the trend towards digital adoption. By embracing this ongoing evolution and leveraging the myriad digital tools, platforms, and technologies at their disposal, businesses can adapt to the ever-changing landscape and position themselves for sustained success in the digital age.

Author: Annie Qureshi

Source: Datafloq

An overview of Morgan Stanley's surge toward data quality

Jeff McMillan, chief analytics and data officer at Morgan Stanley, has long worried about the risks of relying solely on data. If the data put into an institution's system is inaccurate or out of date, it will give customers the wrong advice. At a firm like Morgan Stanley, that just isn't an option.

As a result, Morgan Stanley has been overhauling its approach to data. Chief among them is that it wants to improve data quality in core business processing.

“The acceleration of data volume and the opportunity this data presents for efficiency and product innovation is expanding dramatically,” said Gerard Hester, head of the bank’s data center of excellence. “We want to be sure we are ahead of the game.”

The data center of excellence was established in 2018. Hester describes it as a hub with spokes out to all parts of the organization, including equities, fixed income, research, banking, investment management, wealth management, legal, compliance, risk, finance and operations. Each division has its own data requirements.

“Being able to pull all this data together across the firm we think will help Morgan Stanley’s franchise internally as well as the product we can offer to our clients,” Hester said.

The firm hopes that improved data quality will let the bank build higher quality artificial intelligence and machine learning tools to deliver insights and guide business decisions. One product expected to benefit from this is the 'next best action' the bank developed for its financial advisers.

This next best action uses machine learning and predictive analytics to analyze research reports and market data, identify investment possibilities, and match them to individual clients’ preferences. Financial advisers can choose to use the next best action’s suggestions or not.

Another tool that could benefit from better data is an internal virtual assistant called 'ask research'. Ask research provides quick answers to routine questions like, “What’s Google’s earnings per share?” or “Send me your latest model for Google.” This technology is currently being tested in several departments, including wealth management.

New data strategy

Better data quality is just one of the goals of the revamp. Another is to have tighter control and oversight over where and how data is being used, and to ensure the right data is being used to deliver new products to clients.

To make this happen, the bank recently created a new data strategy with three pillar. The first is working with each business area to understand their data issues and begin to address those issues.

“We have made significant progress in the last nine months working with a number of our businesses, specifically our equities business,” Hester said.

The second pillar is tools and innovation that improve data access and security. The third pillar is an identity framework.

At the end of February, the bank hired Liezel McCord to oversee data policy within the new strategy. Until recently, McCord was an external consultant helping Morgan Stanley with its Brexit strategy. One of McCord’s responsibilities will be to improve data ownership, to hold data owners accountable when the data they create is wrong and to give them credit when it’s right.

“It’s incredibly important that we have clear ownership of the data,” Hester said. “Imagine you’re joining lots of pieces of data. If the quality isn’t high for one of those sources of data, that could undermine the work you’re trying to do.”

Data owners will be held accountable for the accuracy, security and quality of the data they contribute and make sure that any issues are addressed.

Trend of data quality projects

Arindam Choudhury, the banking and capital markets leader at Capgemini, said many banks are refocusing on data as it gets distributed in new applications.

Some are driven by regulatory concerns, he said. For example, the Basel Committee on Banking Supervision's standard number 239 (principles for effective risk data aggregation and risk reporting) is pushing some institutions to make data management changes.

“In the first go-round, people complied with it, but as point-to-point interfaces and applications, which was not very cost effective,” Choudhury said. “So now people are looking at moving to the cloud or a data lake, they’re looking at a more rationalized way and a more cost-effective way of implementing those principles.”

Another trend pushing banks to get their data house in order is competition from fintechs.

“One challenge that almost every financial services organization has today is they’re being disintermediated by a lot of the fintechs, so they’re looking at assets that can be used to either partner with these fintechs or protect or even grow their business,” Choudhury said. “So they’re taking a closer look at the data access they have. Organizations are starting to look at data as a strategic asset and try to find ways to monetize it.”

A third driver is the desire for better analytics and reports.

"There’s a strong trend toward centralizing and figuring out, where does this data come from, what is the provenance of this data, who touched it, what kinds of rules did we apply to it?” Choudhury said. That, he said, could lead to explainable, valid and trustworthy AI.

Author: Penny Crosman

Source: Information-management

Applying data science to battle childhood cancer

Acute myeloid leukaemia in children has a poor prognosis and treatment options unchanged for decades. One collaboration is using data analytics to bring a fresh approach to tackling the disease.

Acute myeloid leukaemia (AML) kills hundreds of children a year. It's the type of cancer that causes the most deaths in children under two, and in teenagers. It has a poor prognosis, and its treatments can be severely toxic.

Research initiative Target Paediatric AML (tpAML) was set up to change the way that the disease is diagnosed, monitored and treated, through greater use of personalised medicine. Rather than the current one-size-fits-all approach for many diseases, personalised medicine aims to tailor an individual's treatment by looking at their unique circumstance, needs, health, and genetics.

AML is caused by many different types of genetic mutation, alone and together. Those differences can affect how the cancer should be treated and its prognosis. To understand better how to find, track and treat the condition, tpAML researchers began building the largest dataset ever compiled around the disease. By sequencing the genomes of over 2,000 people, both alive and deceased, who had the disease, tpAML's researchers hoped to find previously unknown links between certain mutations and how a cancer could be tackled.

Genomic data is notoriously sizeable, and tpAML's sequencing had generated over a petabyte of it. As well as difficulties thrown up by the sheer bulk of data to be analysed, tpAML's data was also hugely complex: each patient's data had 48,000 linked RNA transcripts to analyse.

Earlier this year, Joe Depa, a father who had lost a daughter to the disease and was working with tpAML, joined with his coworkers at Accenture to work on a project to build a system that could analyse the imposing dataset.

Linking up with tpAML's affiliated data scientists and computational working group, Depa along with data-scientist and genomic-expert colleagues hoped to help turn the data into information that researchers and clinicians could use in the fight against paediatric AML, by allowing them to correlate what was happening at a genetic level with outcomes in the disease.

In order to turn the raw data into something that could generate insights into paediatric AML, Accenture staff created a tool that ingested the raw clinical and genomic data and cleaned it up, so analytics tools could process it more effectively. Using Alteryx and Python, the data was merged into a single file, and any incomplete or duplicate data removed. Python was used to profile the data and develop statistical summaries for the analysis – which could be used to flag genes that could be of interest to researchers, Depa says. The harmonised DataFrame was exported as a flat file for more analysis.

"The whole idea was 'let's reduce the time for data preparation', which is a consistent issue in any area around data, but particularly in the clinical space. There's been a tonne of work already put into play for this, and now we hope we've got it in a position where hopefully the doctors can spend more time analysing the data versus having to clean up the data," says Depa, managing director at Accenture Applied Intelligence.

Built using R, the code base that was created for the project is open source, allowing researchers and doctors with similar challenges, but working on different conditions, to reuse the group's work for their own research. While users may need a degree of technical expertise to properly manipulate the information at present, the group is working on a UI that should make it as accessible as possible for those who don't have a similar background.

"We wanted to make sure that at the end of this analysis, any doctor in the world can access this data, leverage this data and perform their analysis on it to hopefully drive to more precision-type medicine," says Depa.

But clinical researchers and doctors aren't always gifted data scientists, so the group has been working on ways to visualise the information, using Unity. The tools they've created allow researchers to manipulate the data in 3D, and zoom in and out on anomalies in the data to find data points that may be worthy of further exploration. One enterprising researcher has even been able to explore those datasets in virtual reality using an Oculus.

Historically, paediatric and adult AML were treated as largely the same disease. However, according to Dr Soheil Meshinchi, professor in the Fred Hutchinson Cancer Research Center's clinical research division and lead for tpAML's computational working group, the two groups stem from different causes. In adults, the disease arises from changes to the smallest links in the DNA chain, known as single base pairs, while in children it's driven by alterations to larger chunks of their chromosomes.

The tpAML has allowed researchers to find previously unknown alterations that cause the disease in children. "We've used the data that tpAML generated to probably make the most robust diagnostic platform that there is. We've identified genetic alterations which was not possible by conventional methods," says Meshinchi.

Once those mutations are found, the data analysis platformcan begin identifying drugs that could potentially target them. Protocols for how to treat paediatric AML have remained largely unchanged for decades and new, more individualised treatment options are sorely needed.

"We've tried it for 40 years of treating all AML the same and hoping for the best. That hasn't worked – you really need to take a step back and to treat each subset more appropriately based on the target that's expressed," says Meshinchi.

The data could help by identifying drugs that have already been developed to treat other conditions but may have a role in fighting paediatric AML, and by showing the pharmaceutical companies that make those drugs there is hard evidence that starting the expensive and risky.

Using the analytics platform to find drugs that can be repurposed in this way, rather than created from scratch, could cut the time it takes for a new paediatric AML treatment to be approved by years. One drug identified as a result has already been tested in clinical trials.

The results generated by the team's work has begun to have an impact for paediatric AML patients. When the data was used to show a subset of children with the disease who had a particular genetic marker that were considered particularly high risk, the treatment pathway for those children was altered.

"This data will not only have an impact ongoing but is already having an impact right now," says Julie Guillot, co-founder of tpAML.

"One cure for leukaemia or one cure for AML is very much unlikely. But we are searching for tailored treatments for specific groups of kids… when [Meshinchi] and his peers are able to find that Achilles heel for a specific cluster of patients, the results are dramatic. These kids go from a very low percentage of cure to, for example, a group that went to 95%. This approach can actually work."

Author: Jo Best

Source: ZDNet

Becoming a better data scientist by improving your SQL skills

Learning advanced SQL skills can help data scientists effectively query their databases and unlock new insights into data relationships, resulting in more useful information.

The skills people most often associate with data scientists are usually those "hard" technical and math skills, including statistics, probability, linear algebra, algorithm knowledge and data visualization.  They need to understand how to work with structured and unstructured data stores and use machine learning and analytics programs to extract valuable information from these stores.

Data scientists also need to possess "soft" skills such as business and domain process knowledge, problem solving, communication and collaboration.

These skills, combined with advanced SQL abilities, enable data scientists to extract value, information and insight from data.

In order to unlock the full value from data, data scientists need to have a collection of tools for dealing with structured information. Many organizations still operate and rely heavily on structured enterprise data stores, data warehouses and databases. Having advanced skills to extract, manipulate and transform this data can really set data scientists apart from the pack.

Advanced vs. beginner SQL skills for data scientists

The common tool and language for interacting with structured data stores is the Structured Query Language (SQL), a standard, widely adopted syntax for data stores that contain schemas that define the structure of their information. SQL allows the user to query, manipulate, edit, update and retrieve data from data sources, including the relational database, an omnipresent feature of modern enterprises.

Relational databases that utilize SQL are popular within organizations, so data scientists should have SQL knowledge at both the basic and advanced levels.

Basic SQL skills include knowing how to extract information from data tables as well as how to insert and update those records.

Because relational databases are often large with many columns and millions of rows, data scientists won't want to pull the entire database for most queries but rather extract only the information needed from a table. As a result, data scientists will need to know at a fundamental level how to apply conditional filters to filter and extract only the data they need.

For most cases, the data that analysts need to work with will not live on just one database, and certainly not in a single table in that database.

It's not uncommon for organizations to have hundreds or thousands of tables spread across hundreds or thousands of databases that were created by different groups and at different periods. Data scientists need to know how to join these multiple tables and databases together, making it easier to analyze different data sets.

So, data scientists need to have deep knowledge of JOIN and SELECT operations in SQL as well as their impact on overall query performance.

However, to address more complex data analytics needs, data scientists need to move beyond these basic skills and gain advanced SQL skills to enable a wider range of analytic abilities. These advanced skills enable data scientists to work more quickly and efficiently with structured databases without having to rely on data engineering team members or groups.

Understanding advanced SQL skills can help data scientists stand out to potential employers or shine internally.

Types of advanced SQL skills data scientists need to know

Advanced SQL skills often mean distributing information across multiple stores, efficiently querying and combining that data for specific analytic purposes.

Some of these skills include the following:

Advanced and nested subqueries. Subqueries and nested queries are important to combine and link data between different sources. Combined with advanced JOIN operations, subqueries can be faster and more efficient than basic JOIN or queries because they eliminate extra steps in data extraction.

Common table expressions. Common table expressions allow you to create a temporary table that enables temporary storage while working on large query operations. Multiple subqueries can complicate things, so table expressions help you break down your code into smaller chunks, making it easier to make sense of everything. 

Efficient use of indexes. Indexes keep relational databases functioning effectively by setting up the system for expecting and optimizing for particular queries. Efficient use of indexes can greatly speed up performance, making data easier and faster to find. Conversely, poor use of indexing can lead to high query time and slow query performance, resulting in systems that can have runaway performance when queried at scale.

Advanced use of date and time operations. Knowing how to manipulate date and time can come in handy, especially when working with time-series data. Advanced date operations might require knowledge of date parsing, time formats, date and time ranges, time grouping, time sorting and other activities that involve the use of timestamps and date formatting.

Delta values. For many reasons, you may want to compare values from different periods. For example, you might want to evaluate sales from this month versus last month or sales from December this year versus December last year. You can find the difference between these numbers by running delta queries to uncover insights or trends you may not have seen otherwise.

Ranking and sorting methods. Being able to rank and sort rows or values is necessary to help uncover key insights from data. Data analytics requirements might include ranking data by number of products or units sold, top items viewed, or top sources of purchases. Knowing advanced methods for ranking and sorting can optimize overall query time and provide accurate results.

Query optimization. Effective data analysts spend time not only formulating queries but optimizing them for performance. This skill is incredibly important once databases grow past a certain size or are distributed across multiple sources. Knowing how to deal with complex queries and generate valuable results promptly with optimal performance is a key skill for effective data scientists.

The value of advanced SQL skills

The main purpose of data science is to help organizations derive value by finding information needles in data haystacks. Data scientists need to be masters at filtering, sorting and summarizing data to provide this value. Advanced SQL skills are core to providing this ability.

Organizations are always looking to find data science unicorns who have all the skills they want and more. Knowing different ways to shape data for targeted analysis is incredibly desirable.

For many decades, companies have stored valuable information in relational databases, including transactional data and customer data. Feeling comfortable finding, manipulating, extracting, joining or adding data to these databases will give data scientists a leg up on creating value from this data.

As with any skill, learning advanced SQL skills will take time and practice to master. However, enterprises provide many opportunities for data scientists and data analysts to master those skills and provide more value to the organization with real-life data and business problems to solve.

Author: Kathleen Walch

Source: TechTarget

BERT-SQuAD: Interviewing AI about AI

If you’re looking for a data science job, you’ve probably noticed that the field is hyper-competitive. AI can now even generate code in any language. Below, we’ll explore how AI can extract information from paragraphs to answer questions.

One day you might be competing against AI, if AutoML isn’t that competitor already.

What is BERT-SQuAD?

Google BERT and the Stanford Question Answering Dataset.

BERT is a cutting-edge Natural Language Processing algorithm that can be used for tasks like question answering (which we’ll go into here), sentiment analysis, spam filtering, document clustering, and more. It’s all language!

“Bidirectionality” refers to the fact that many words change depending on their context, like “let’s hit he club” versus “an idea hit him”, so it’ll consider words on both sides of the keyword.

“Encoding” just means assigning numbers to characters, or turning an input like “let’s hit the club” into a machine-workable format.

“Representations” are the general understanding of words you get by looking at many of their encodings in a corpus of text.

“Transformers” are what you use to get from embeddings to representations. This is the most complex part.

As mentioned, BERT can be trained to work on basically any kind of language task, so SQuAD refers to the dataset we’re using to train it on a specific language task: Question answering.

SQuAD is a reading comprehension dataset, containing questions asked by crowdworkers on Wikipedia articles, where the answer to every question is a segment of text from the corresponding passage.

BERT-SQuAD, then, allows us to answer general questions by fishing out the answer from a body of text. It’s not cooking up answers from scratch, but rather, it understands the context of the text enough to find the specific area of an answer.

For example, here’s a context paragraph about lasso and ridge regression:

“You can quote ISLR’s authors Hastie, Tibshirani who asserted that, in presence of few variables with medium / large sized effect, use lasso regression. In presence of many variables with small / medium sized effect, use ridge regression.

Conceptually, we can say, lasso regression (L1) does both variable selection and parameter shrinkage, whereas Ridge regression only does parameter shrinkage and end up including all the coefficients in the model. In presence of correlated variables, ridge regression might be the preferred choice. Also, ridge regression works best in situations where the least square estimates have higher variance. Therefore, it depends on our model objective.”

Now, we could ask BERT-SQuAD:

“When is Ridge regression favorable over Lasso regression?”

And it’ll answer:

“In presence of correlated variables”

While I show around 100 words of context here, you could input far more context into BERT-SQuAD, like whole documents, and quickly retrieve answers. An intelligent Ctrl-F, if you will.

To test the following 7 questions, I used Gradio, a library that lets developers make interfaces out of models. In this case, I used the BERT-SQuAD interface created out of Google Colab.

I used the contexts from a Kaggle thread as inputs, and modified the questions for simplicities sake.

Q1: What will happen if you don’t rotate PCA components?

The effect of PCA will diminish

Q2. How do you reduce the dimensions of data to reduce computation time?

We can separate the numerical and categorical variables and remove the correlated variables

Q3: Why is Naive Bayes “naive” ?

It assumes that all of the features in a data set are equally important and independent

Q4: Which algorithm should you use to tackle low bias and high variance?

Bagging

Q5: How are kNN and kmeans clustering different?

kmeans is unsupervised in nature and kNN is supervised in nature

Q6: When is Ridge regression favorable over Lasso regression?

In presence of correlated variables

Q7: What is convex hull?

Represents the outer boundaries of the two group of data points

Author: Frederik Bussler

Source: Towards Data Science

BI topics to tackle when migrating to the cloud

When your organization decides to pull the trigger on a cloud migration, a lot of stuff will start happening all at once. Regardless of how long the planning process has been, once data starts being relocated, a variety of competing factors that have all been theoretical earlier become devastatingly real: frontline business users still want to be able to run analyses while the migration is happening, your data engineers are concerned with the switch from whatever database you were using before, and the development org has its own data needs. With a comprehensive, BI-focused data strategy, you and your stakeholders will know what your ideal data model should look like once all your data is moved over. This way, as you’re managing the process and trying to keep everyone happy, you end in a stronger place when your migration is over than you were at the start, and isn’t that the goal?

BI focus and your data infrastructure

“What does all this have to do with my data model?” you might be wondering. “And for that matter, my BI solution?”

I’m glad you asked, internet stranger. The answer is everything. Your data infrastructure underpins your data model and powers all of your business-critical IT systems. The form it takes can have immense ramifications for your organization, your product, and the new things you want to do with it. Your data infrastructure is hooked into your BI solution via connectors, so it’ll work no matter where the data is stored. Picking the right data model, once all your data is in its new home, is the final piece that will allow you to get the most out of it with your BI solution. If you don’t have a BI solution, the perfect time to implement one is once all your data is moved over and your model is built. This should all be part of your organization’s holistic cloud strategy, with buy-in from major partners who are handling the migration.

Picking the right database model for you

So you’re giving your data a new home and maybe implementing a BI solution when it’s all done. Now, what database model is right for your company and your use case? There are a wide array of ways to organize data, depending on what you want to do with it.

One of the broadest is a conceptual model, which focuses on representing the objects that matter most to the business and the relationships between them. This database model is designed principally for business users. Compare this to a physical model, which is all about the structure of the data. In this model, you’ll be dealing with tables, columns, relationships, graphs, etc. And foreign keys, which distinguish the connections between the tables.

Now, let’s say you’re only focused on representing your data organization and architecture graphically, putting aside the physical usage or database management framework. In cases like these, a logical model could be the way to go. Examples of these types of databases include relational (dealing with data as tables or relations), network (putting data in the form of records), and hierarchical (which is a progressive tree-type structure, with each branch of the tree showing related records). These models all feature a high degree of standardization and cover all entities in the dataset and the relationships between them.

Got a wide array of different objects and types of data to deal with? Consider an object-oriented database model, sometimes called a “hybrid model.” These models look at their contained data as a collection of reusable software pieces, all with related features. They also consolidate tables but aren’t limited to the tables, giving you freedom when dealing with lots of varied data. You can use this kind of model for multimedia items you can’t put in a relational database or to create a hypertext database to connect to another object and sort out divergent information.

Lastly, we can’t help but mention the star schema here, which has elements arranged around a central core and looks like an asterisk. This model is great for querying informational indexes as part of a larger data pool. It’s used to dig up insights for business users, OLAP cubes, analytics apps, and ad-hoc analyses. It’s a simple, yet powerful, structure that sees a lot of usage, despite its simplicity.

Now what?

Whether you’re building awesome analytics into your app or empowering in-house users to get more out of your data, knowing what you’re doing with your data is key to maintaining the right models. Once you’ve picked your database, it’s time to pick your data model, with an eye towards what you want to do with it once it’s hooked into your BI solution.

Worried about losing customers? A predictive churn model can help you get ahead of the curve by putting time and attention into relationships that are at risk of going sour. On the other side of the coin, predictive up- and cross-sell models can show you where you can get more money out of a customer and which ones are ripe to deepen your financial relationship.

What about your marketing efforts? A customer segmentation data model can help you understand the buying behaviors of your current customers and target groups and which marketing plays are having the desired effect. Or go beyond marketing with “next-best-action models” that take into account life events, purchasing behaviors, social media, and anything else you can get your hands on so that you can figure out what’s the next action with a given target (email, ads, phone call, etc.) to have the greatest impact. And predictive analyses aren’t just for humancentric activities, manufacturing and logistics companies can take advantage of maintenance models that can let you circumvent machine breakdowns based on historical data. Don’t get caught without a vital piece of equipment again.

Bringing it all together with BI

Staying focused on your long-term goals is an important key to success. Whether you’re building a game-changing product or rebuilding your data model, having a well defined goal makes all the difference in the world when it comes to the success of your enterprise. If you’re already migrating your data to the cloud, then you’re at the perfect juncture to pick the right database and data models for your eventual use cases. Once these are set up, they’ll integrate seamlessly with your BI tool (and if you don’t have one yet, it’ll be the perfect time to implement one). Big moves like this represent big challenges, but also big opportunities to make lay the foundation for whatever you’re planning on building. Then you just have to build it!

Author: Jack Cieslak

Source: Sisense

Big Data Analytics in Banking

Banking institutions need to use big data to remodel customer segmentation into a solution that works better for the industry and its customers. Basic customer segmentation generalizes customer wants and needs without addressing any of their pain points. Big data allows the banking industry to create individualized customer profiles that help decrease the pains and gaps between bankers and their clients. Big data analytics allows banks to examine large sets of data to find patterns in customer behavior and preferences. Some of this data includes social media behavior.

• Demographic information.
• Customer spending.
• Product and service usage — including offers that customers have declined.
• Impactful life events.
• Relationships between bank customers.
• Service preferences and attitudes toward the banking industry as a whole.

Providing a Personalized Customer Experience with Big Data Analytics

Banking isn’t known for being an industry that provides tailor-made customer service experiences. Now, with the combination of service history and customer profiles made available by big data analytics, bank culture is changing. 

Profiling has an invasive ring to it, but it’s really just an online version of what bankers are already doing. Online banking has made it possible for customers to transfer money, deposit checks and pay bills all from their mobile devices. The human interaction that has been traditionally used to analyze customer behavior and create solutions for pain points has gone digital. 

Banks can increase customer satisfaction and retention due to profiling. Big data analytics allows banks to create a more complete picture of what each of their customers is like, not just a generic view of them. It tracks their actual online banking behaviors and tailors its services to their preferences, like a friendly teller would with the same customer at their local branch. 

Artificial Intelligence’s Role in Banking

Nothing will ever beat the customer service you can receive in a conversation with a real human being. But human resources are limited by many physical factors that artificial intelligence (AI) can make up for. Where customer service agents may not be able to respond in a timely manner to customer inquiries depending on demand, AI can step in. 

Chatbots enable customers to receive immediate answers to their questions. Their AI technology uses customer profile information and behavioral patterns to give personalized responses to inquiries. They can even recognize emotions to respond sensitively depending on the customers’ needs. 

Another improvement we owe to AI is simplified online banking. Advanced machine learning accurately pulls information from documents uploaded online and on mobile apps. This technology is the reason why people can conveniently deposit checks from their smartphones. 

Effective Fraud Prevention

Identity fraud is one of the fastest growing forms of theft. With more than 16 million identity theft cases in 2017, fraud protection is becoming increasingly important in the banking industry. Big data analytics can help banks in securing customer account information.

Business intelligence (BI) tools are used in banking to evaluate risk and prevent fraud. The big data retrieved from these tools determines interest rates for individuals, finds credit scores and pinpoints fraudulent behavior. Big data that’s analyzed to find market trends can help inform personal and industry-wide financial decisions, such as increasing debt monitoring rates.

Similarly, using big data for predictive purposes can also help financial institutions avoid financial crises before they happen by collecting information on things like cross-border debt and debt-service ratios.

The Future of Big Data Analytics

The banking industry can say goodbye to their outdated system of customer guesswork. Big data analytics have made it possible to monitor the financial health and needs of customers, including small business clients. 

Banks can now leverage big data analytics to detect fraud and assess risks, personalize banking services and create AI-driven customer resources. Data volume will only continue to increase with time as more people create and use this information. The mass of information will grow, but so will its profitability as more industries adopt big data analytic tools. 

Big data will continue to aid researchers in discovering market trends and making timely decisions. The internet has changed the way people think and interact, which is why the banking industry must utilize big data to keep up with customer needs. As technology continues to improve at a rapid pace, any business who falls behind may be left there.

Author: Shannon Flynn

Source: Open Data Science

Er gaat momenteel geen dag voorbij of er is in de media wel een bericht of discussie te vinden rond data. Of het nu gaat om vraagstukken rond privacy, nieuwe mogelijkheden en bedreigingen van Big Data, of nieuwe diensten gebaseerd op het slim combineren en uitwisselen van gegevens: je kunt er niet onderuit dat informatie ‘hot’ is. 

Is Big Data Analytics - ofwel de analyse van grote hoeveelheden data, veelal ongestructureerd - een hype? Toen de term enkele jaren geleden opeens overal opdook zeiden veel sceptici dat het een truc was van software leveranciers om iets bestaands - data analyse wordt al lang toegepast - opnieuw te vermarkten. Inmiddels zijn alle experts het er over eens dat Big Data Analytics in de vorm waarin het nu kan worden toegepast een enorme impact gaat hebben op de wereld zoals wij die kennen. Ja, het is een hype, maar wel een terechte.

Big Data Analytics – wat is dat nou eigenlijk?

Big Data is al jaren een hype, en zal dat nog wel even blijven. Wanneer is er nou sprake van ‘Big’ Data, bij hoeveel tera-, peta- of yottabytes (1024) ligt de grens tussen ‘Normal’ en ‘Big’ Data? Het antwoord is: er is geen duidelijke grens. Je spreekt van Big Data als het te veel wordt voor jouw mensen en middelen. Big Data Analytics richt zich op de exploratie van data middels statistische methoden om nieuwe inzichten op te doen waarmee de toekomstige prestaties verbeterd kunnen worden. 

Big Data Analytics als stuurmiddel voor prestaties is al volop in gebruik bij bedrijven. Denk aan een sportclub die het inzet om te bepalen welke spelers ze gaan kopen. Of een bank die gestopt is alleen talenten te rekruteren van topuniversiteiten omdat bleek dat kandidaten van minder prestigieuze universiteiten het beter deden. Of bijvoorbeeld een verzekeringsmaatschappij die het gebruikt om fraude te detecteren. Enzovoorts. Enzovoorts. 

Wat maakt Big Data Analytics mogelijk? 

Tenminste drie ontwikkelingen zorgen ervoor dat Big Data Analytics een hele nieuwe fase ingaat. 

1. Rekenkracht 

De toenemende rekenkracht van computers stelt analisten in staat om enorme datasets te gebruiken, en een groot aantal variabelen te gebruiken in hun analyses. Door de toegenomen rekenkracht is het niet langer nodig om een steekproef te nemen zoals vroeger, maar kan alle data gebruikt worden voor een analyse. De analyse kan worden gedaan met behulp van specifieke tools en vereist vaak specifieke kennis en vaardigheden van de gebruiker, een data analist of data scientist. 

2. Datacreatie 

Het internet en social media zorgen ervoor dat de hoeveelheid data die we creëren exponentieel toeneemt. Deze data is inzetbaar voor talloze data-analyse toepassingen, waarvan de meeste nog bedacht moeten worden. 

Om een beeld te krijgen van de datagroei, overweeg deze statistieken: 

- Meer dan een miljard tweets worden iedere 48 uur verstuurd.

- Dagelijks komen een miljoen Twitter accounts bij.

- Iedere 60 seconden worden er 293.000 status updates gepost op facebook.

- De gemiddelde Facebook gebruiker creëert 90 stukken content per maand, inclusief links, nieuws, verhalen, foto’s en video’s. 

- Elke minuut komen er 500 Facebook accounts bij. 

- Iedere dag worden 350 miljoen foto’s geupload op facebook, wat neerkomt op 4.000 foto’s per seconde.

- Als Wikipedia een boek zou zijn, zou het meer dan twee miljard pagina’s omvatten. 

Bron: http://www.iacpsocialmedia.org

3. Dataopslag 

De kosten voor het opslaan van data zijn sterk afgenomen de afgelopen jaren, wat de mogelijkheden om analytics toe te passen heeft doen groeien. Een voorbeeld is de opslag van videobeelden. Beveiligingscamera’s in een supermarkt namen eerst alles op tape op. Als er na drie dagen niks gebeurd was werd de band teruggespoeld en werd er opnieuw over opgenomen.  

Dat is niet langer nodig. Een supermarkt kan nu digitale beelden - die de hele winkel vastleggen - naar de cloud versturen waar ze blijven opgeslagen. Vervolgens is het mogelijk analytics op deze beelden toe te passen: welke promoties werken goed? Voor welke schappen blijven mensen lang staan? Wat zijn de blinde hoeken in de winkel? Of predictive analytics: Stel dat we dit product in dit schap zouden leggen, wat zou het resultaat dan zijn? Deze analyses kan het management gebruiken om tot een optimale winkelinrichting te komen en maximaal rendement uit promoties te halen.  

Betekenis Big Data Analytics

Big Data - of Smart Data - zoals Bernard Marr, auteur van het nieuwe praktische boek ‘Big Data: Using SMART Big Data Analytics To Make Better Decisions and Improve Performance’ - het liever noemt is de wereld aan het veranderen. De hoeveelheid data neemt exponentieel toe momenteel, maar de hoeveelheid is voor de meeste beslissers grotendeels irrelevant. Het gaat erom hoe men het inzet om te komen tot waardevolle inzichten.  

Big Data 

De meningen zijn verdeeld over wat big data nou precies is. Gartner definieert big data vanuit de drie V’s Volume, Velocity en Variety. Het gaat dus om de hoeveelheid data, de snelheid waarmee de data verwerkt kan worden en de diversiteit van de data. Met dit laatste wordt bedoeld dat de data, naast gestructureerde bronnen, ook uit allerlei ongestructureerde bronnen gehaald kan worden, zoals internet en social media, inclusief tekst, spraak en beeldmateriaal.

Analytics

Wie zou niet de toekomst willen voorspellen? Met voldoende data, de juiste technologie en een dosis wiskunde komt dat binnen bereik. Dit wordt business analytics genoemd, maar er zijn veel andere termen in omloop, zoals data science, machine learning en, jawel, big data. Ondanks dat deze wiskunde al vrij lang bestaat, is het nog een relatief nieuw vakgebied dat tot voor kort alleen voor gespecialiseerde bedrijven met veel geld bereikbaar was.

Toch maken we er zonder het te weten allemaal al gebruik van. Spraakherkenning op je telefoon, virusscanners op je PC en spamfilters voor email zijn gebaseerd op concepten die in het domein van business analytics vallen. Ook de ontwikkeling van zelfrijdende auto’s en alle stapjes daarnaartoe (adaptive cruise control, lane departure system, et cetera) zijn alleen mogelijk door machine learning. 

Analytics is kortom de ontdekking en de communicatie van zinvolle patronen in data. Bedrijven kunnen analytics toepassen op zakelijke gegevens om hun bedrijfsprestaties te beschrijven, voorspellen en verbeteren. Er zijn verschillende soorten analytics, zoals tekst-analytics, spraak-analytics en video-analytics. 

Een voorbeeld van tekst-analytics is een advocatenfirma die hiermee duizenden documenten doorzoekt om zo snel de benodigde informatie te vinden ter voorbereiding van een nieuwe zaak. Speech-analytics worden bijvoorbeeld gebruikt in callcenters om vast te stellen wat de stemming van de beller is, zodat de medewerker hier zo goed mogelijk op kan anticiperen. Video-analytics kan gebruikt worden voor het monitoren van beveiligingscamera’s. Vreemde patronen worden er zo uitgepikt, waarop beveiligingsmensen in actie kunnen komen. Ze hoeven nu zelf niet langer uren naar het scherm te staren terwijl er niks gebeurt.  

Het proces kan zowel top-down als bottom-up benaderd worden. De meest toegepaste benaderingen zijn: 

• Datamining: Dataonderzoek op basis van een gerichte vraag, waarin men op zoek gaat naar een specifiek antwoord.
• Trend-analyse en predictive analytics: Door gericht op zoek te gaan naar oorzaak-gevolg verbanden om bepaalde gebeurtenissen te kunnen verklaren of om toekomstig gedrag te voorspellen.
• Data discovery: Data onderzoeken op onverwachte verbanden of andere opvallende zaken.

Feiten en dimensies

De data die helpen om inzichten te verkrijgen of besluiten te nemen zijn feiten. Bijvoorbeeld EBITDA, omzet of aantal klanten. Deze feiten krijgen waarde door dimensies. De omzet over het jaar 2014 voor de productlijn babyvoeding in de Regio Oost. Door met dimensies te gaan analyseren kun je verbanden ontdekken, trends benoemen en voorspellingen doen voor de toekomst.

Analytics versus Business Intelligence

Waarin verschilt analytics nu van business intelligence (BI)? In feite is analytics op data gebaseerde ondersteuning van de besluitvorming. BI toont wat er gebeurd is op basis van historische gegevens die gepresenteerd worden in vooraf bepaalde rapporten. Waar BI inzicht geeft in het verleden, focust analytics zich op de toekomst. Analytics vertelt wat er kan gaan gebeuren door op basis van de dagelijks veranderende datastroom met ‘wat als’- scenario’s inschattingen te maken en risico’s en trends te voorspellen.

Voorbeelden Big Data Analytics

De wereld wordt steeds slimmer. Alles is meetbaar, van onze hartslag tijdens een rondje joggen tot de looppatronen in winkels. Door die data te gebruiken, kunnen we indrukwekkende analyses maken om bijvoorbeeld filevorming te voorkomen, epidemieën voortijdig te onderdrukken en medicijnen op maat aan te bieden.

Deze evolutie is zelfs zichtbaar in de meest traditionele industrieën, zoals de visserij. In plaats van - zoals vanouds - puur te vertrouwen op een kompas en ‘insider knowledge’ doorgegeven door generaties vissersfamilies, koppelt de hedendaagse visser sensoren aan vissen en worden scholen opgespoord met de meest geavanceerde GPS-systemen. Big Data Analytics wordt inmiddels toegepast in alle industrieën en sectoren. Ook steden maken er gebruik van. Hieronder een overzicht van mogelijke toepassingen:

Doelgroep beter begrijpen

De Amerikaanse mega retailer Target weet door een combinatie van 25 aankopen wanneer een vrouw zwanger is. Dat is één van de weinige perioden in een mensenleven waarin koopgedrag afwijkt van routines. Hier speelt Target slim op in met baby-gerelateerde aanbiedingen. Amazon is zo goed geworden in predictive analytics dat ze producten al naar naar je toe kunnen sturen voordat je ze gekocht hebt. Als het aan hun ligt, kun je je bestelling binnenkort middels een drone binnen 30 minuten bezorgd krijgen.

Processen verbeteren 

Processen veranderen ook door Big Data. Bijvoorbeeld inkoop. Walmart weet dat er meer ‘Pop Tarts’ verkocht worden bij een stormwaarschuwing. Ze weten niet waarom dat is, maar ze zorgen er wel voor dat ze voldoende voorraad hebben en de snacks een mooie plek in de winkel geven. Een ander proces waar data grote kansen biedt voor optimalisatie is de supply chain. Welke routes laat je chauffeurs rijden en in welke volgorde laat je ze bestellingen afleveren? Real-time weer- en verkeerdata zorgt voor bijsturing. 

Business optimalisatie

Bij Q-Park betalen klanten per minuut voor parkeren, maar het is ook mogelijk een abonnement af te nemen. De prijs per minuut is bij een abonnement vele malen goedkoper. Als de garage vol begint te raken, is het vervelend als er net een klant met abonnement aan komt rijden, want dat kost omzet. Het analytics systeem berekent daarom periodiek de optimale mix van abonnementsplekken en niet abonnementsplekken op basis van historische gegevens. Zo haalt de garage exploitant het maximale eruit wat eruit te halen valt. 

Optimalisatie machines 

General Electric (GE) is een enthousiast gebruiker van big data. Het conglomeraat gebruikt al veel data in haar data-intensieve sectoren, zoals gezondheidszorg en financiële dienstverlening, maar het bedrijf ziet ook industriële toepassingen, zoals in GE’s businesses voor locomotieven, straalmotoren en gasturbines. GE typeert de apparaten in bedrijfstakken als deze ook wel als ‘dingen die draaien’ en verwacht dat de meeste van die dingen, zo niet alle, binnenkort gegevens over dat ‘draaien’ kunnen vastleggen en communiceren. 

Een van die draaiende dingen is de gasturbine die de klanten van GE gebruiken voor energieopwekking. GE monitort nu al meer dan 1500 turbines vanuit een centrale faciliteit, dus een groot deel van de infrastructuur voor gebruik van big data om de prestaties te verbeteren is er al. GE schat dat het de efficiëntie van de gemonitorde turbines met minstens 1 procent kan verbeteren via software en netwerkoptimalisatie, doeltreffender afhandelen van onderhoud en betere harmonisering van het gas-energiesysteem. dat lijkt misschien niet veel, maar het zou neerkomen op een brandstofbesparing van 66 miljard dollar in de komende 15 jaar.
(bron: 'Big Data aan het werk' door Thomas Davenport)

Klantenservice en commercie

Een grote winst van de nieuwe mogelijkheden van big data voor bedrijven is dat ze alles aan elkaar kunnen verbinden; silo’s, systemen, producten, klanten, enzovoorts. Binnen de telecom hebben ze bijvoorbeeld het cost-to-serve-concept geïntroduceerd. Daarmee kunnen zij vanuit de daadwerkelijke operatie kijken wat voor contactpunten ze met de klant hebben; hoe vaak hij belt met de klantenservice; wat zijn betaalgedrag is; hoe hij zijn abonnement gebruikt; hoe hij is binnengekomen; hoe lang hij klant is; waar hij woont en werkt; welke telefoon hij gebruikt; et cetera. 

Wanneer het telecombedrijf de data van al die invalshoeken bij elkaar brengt, ontstaat er opeens een hele andere kijk op de kosten en omzet van die klant. In die veelheid van gezichtspunten liggen mogelijkheden. Alleen al door data te integreren en in context te bekijken, ontstaan gegarandeerd verrassende nieuwe inzichten. Waar bedrijven nu typisch naar kijken is de top 10 klanten die het meeste en minste bijdragen aan de omzet. Daar trekken ze dan een streep tussen. Dat is een zeer beperkte toepassing van de beschikbare data. Door de context te schetsen kan het bedrijf wellicht acties bedenken waarmee ze de onderste 10 kunnen enthousiasmeren iets meer te doen. Of er alsnog afscheid van nemen, maar dan weloverwogen.

Slimme steden

New York City maakt tegenwoordig gebruik van een ‘soundscape’ van de hele stad. Een verstoring in het typische stadsgeluid, zoals bijvoorbeeld een pistoolschot, wordt direct doorgegeven aan de politie die er op af kunnen. Criminelen gaan een moeilijke eeuw tegemoet door de toepassing van dergelijke Big Data Analytics. 

Slimme ziekenhuizen

Of het nu gaat om de informatie die gedurende een opname van een patiënt wordt verzameld of informatie uit de algemene jaarrapporten: Big Data wordt voor ziekenhuizen steeds belangrijker voor verbeterde patiëntenzorg, beter wetenschappelijk onderzoek en bedrijfsmatige informatie. Medische data verdubbelen iedere vijf jaar in volume. Deze gegevens kunnen van grote waarde zijn voor het leveren van de juiste zorg.

HR Analytics

Data kan worden aangewend om de prestaties van medewerkers te monitoren en te beoordelen. Dit geldt niet alleen voor de werknemers van bedrijven, maar zal ook steeds vaker worden toegepast om de toplaag van managers en leiders objectief te kunnen beoordelen. 

Een bedrijf dat de vruchten heeft geplukt van HR Analytics is Google. De internet- en techgigant had nooit het geloof dat managers veel impact hadden, dus ging het analyticsteam aan de slag met de vraag: ‘Hebben managers eigenlijk een positieve impact bij Google?’ Hun analyse wees uit dat managers wel degelijk verschil maken en een positieve impact kunnen hebben bij Google. De volgende vraag was: ‘Wat maakt een geweldige manager bij Google?’ Dit resulteerde in 8 gedragingen van de beste managers en de 3 grootste valkuilen. Dit heeft geleid tot een zeer effectief training en feedback programma voor managers dat een hele positieve invloed heeft gehad op de performance van Google.  

Big Data Analytics in het MKB

Een veelgehoorde misvatting over Big Data is dat het alleen iets is voor grote bedrijven. Fout, want ieder bedrijf van groot naar klein kan data inzetten. Bernard Marr geeft in zijn boek een voorbeeld van een kleine mode retail onderneming waar hij mee samen heeft gewerkt. 

De onderneming in kwestie wilden hun sales verhogen. Ze hadden alleen geen data om dit doel te bereiken op de traditionele sales data na. Ze bedachten toen eerst een aantal vragen:

- Hoeveel mensen passeren onze winkels?

- Hoeveel stoppen er om in de etalage te kijken en voor hoe lang?

- Hoeveel komen vervolgens binnen?

- Hoeveel kopen dan iets? 

Vervolgens hebben ze een klein discreet apparaat achter het raam geplaatst dat het aantal passerende mobiele telefoons (en daarmee mensen) is gaan meten. Het apparaat legt ook vast hoeveel mensen voor de etalage blijven staan en voor hoe lang, en hoeveel er naar binnen komen. Sales data legt vervolgens vast hoeveel mensen wat kopen. De winkelketen kon vervolgens experimenteren met verschillende etalages om te testen welke het meest succesvol waren. Dit project heeft geleid tot fors meer omzet, en het sluiten van één worstelend filiaal waar onvoldoende mensen langs bleken te komen.  

Conclusie

De Big Data revolutie maakt de wereld in rap tempo slimmer. Voor bedrijven is de uitdaging dat deze revolutie plaatsvindt naast de ‘business as usual’. Er is nog veel te doen voordat de meeste ondernemingen in staat zijn echt te profiteren van Big Data Analytics. Het gros van de organisaties is al blij dat ze op een goede manier kunnen rapporteren en analyseren. Veel bedrijven moeten nog aan het experiment beginnen, iets waarbij ze mogelijk over hun koudwatervrees heen moeten stappen. Het is in ieder geval zeker dat er nu snel heel veel kansen zullen ontstaan. De race die nu begonnen is zal uitwijzen wie er met de nieuwe inzichten aan de haal gaan. 

Auteur: Jeppe Kleyngeld

Bron: FMI

Big data and the future of the self-driving car

Each year, car manufacturers get closer to successfully developing a fully autonomous vehicle. Over the last several years, major tech companies have paired up with car manufacturers to develop the advanced technology that will one day allow the majority of vehicles on the road to be autonomous. Of the five levels of automation, companies like Ford and Tesla are hovering around level three, which offers several autonomous driving functions but still requires a person to be attentive behind the wheel.

However, car manufacturers are expected to release fully automatic vehicles to the public within the next decade. These vehicles are expected to have a large number of safety and environmental benefits. Self-driving technology has come a long way over the last few years, as the growth of big data in technology industries has helped provide car manufacturers with the programming data needed to get closer to fully automating cars. Big data is helping to install enough information and deep learning in autonomous cars to make them safer for all drivers.

History of self-driving cars

The first major automation in cars was cruise control, which was patented in 1950 and is used by most drivers to keep their speed steady during long drives nowadays. Most modern cars already have several automated functions, like proximity warnings and steering adjustment, which have been tried and tested, and proven to be valuable features for safe driving. These technologies use sensors to alert the driver when they are coming too close to something that may be out of the driver’s view or something that the driver may simply not have noticed.

The fewer functions drivers have to worry about and pay attention to, the more they’re able to focus on the road in front of them and stay alert to dangerous circumstances that could occur at any moment. Human error causes 90 percent of all crashes on the roads, which is one of the main reasons so many industries support the development of autonomous vehicles. However, even when a driver is completely attentive, circumstances that are out of their control could cause them to go off the road or crash into other vehicles. Car manufacturers are still working on the programming for autonomous driving in weather that is less than ideal.

Big data’s role in autonomous vehicle development

Although these technologies provided small steps toward automation, they remained milestones away from a fully automated vehicle. However, over the last decade, with the large range of advancements that have been made in technology and the newfound use of big data, tech companies have discovered the necessary programming for fully automating vehicles. Autonomous vehicles rely entirely on the data they receive through GPS, radar and sensor technology, and the information they process through cameras.

The information cars receive through these sources provides them with the data needed to make safe driving decisions. Although car manufacturers are still using stores of big data to work out the kinks of the thousands of scenarios an autonomous car could find itself in, it’s only a matter of time before self-driving cars transform the automotive industry by making up the majority of cars on the road. As the price of the advanced radars for these vehicles goes down, self-driving cars should become more accessible to the public, which will increase the safety of roads around the world.

Big data is changing industries worldwide, and deep learning is contributing to the progress towards fully autonomous vehicles. Although it will still be several decades before the mass adoption of self-driving cars, the change will slowly but surely come. In only a few decades, we’ll likely be living in a time where cars are a safer form of transportation, and accidents are tragedies that are few and far between.

Source: Insidebigdata

Big Tech: the battle for our data

The most important sector of tech is user privacy and with it comes a war not fought in the skies or trenches but in congressional hearings and slanderous advertisements, this battle fought in the shadows for your data and attention is now coming to light.

The ever-growing reliance we have on technology has boomed since the advent of social media, especially and specifically with phones. Just 15 years ago, the favoured way of accessing services like Facebook was through a computer but this changed at a radical pace following the introduction of the iPhone in 2007 and the opening of the iOS App Store in 2008.

Since then, the app economy now in its teens has become a multi-billion dollar industry built on technologies founded in behavioural change and habit forming psychology.

If you don’t have the iPhone’s ‘Screen Time’ feature set up, you’ll want to do that after hearing this:

According to various studies a typical person spends over four hours a day on their phone, with almost half of that time taken up by social media platforms like Facebook, Instagram, and Twitter. These studies were conducted before the pandemic so it wouldn’t be far stretched to assume these figures have gone up.

So what happens with all this time spent on these platforms?

Your time is your attention, your attention is their data

Where advertisements of old for businesses and products relied on creativity and market research on platforms like television and newspapers, modern advertising takes advantage of your online behaviour and interests to accurately target tailored advertisements to users.

User data collected by Facebook is used to create targeted advertisements for all kinds of products, businesses and services. They use information like your search history, previous purchases, location data and even collect identifying information across apps and websites owned by other companies to build a profile that’s used to advertise things to you. In a recent update to iOS, Apple’s App Store now requires developers to outline to users what data is tracked and collected in what they are calling ‘privacy nutrition labels’.

In response to this in Facebook’s most recent quarterly earnings call, Mark Zuckerberg stated “We have a lot of competitors who make claims about privacy that are often misleading,” and “Now Apple recently released so-called (privacy) nutrition labels, which focused largely on metadata that apps collect rather than the privacy and security of people’s actual messages,”.

Facebook uses this meta-data to sell highly targeted ad space.

This is how you pay for ‘free’ services, with your data and attention

The harvesting of user data on platforms like Facebook has not only benefited corporations in ‘Big Tech’ and smaller business but has even been grossly abused by politicians to manipulate outcomes of major political events.

In 2018, the Cambridge Analytica scandal emerged into the forefront of mainstream media after a whistleblower for the company, Christopher Wylie came forward with information that outlined the unethical use of Facebook user data to create highly targeted advertisements with the goal of swaying political agendas. Most notably, illicitly obtained data was used in former US President Donald Trump’s 2016 presidential campaign in the United States, as well as the Leave. EU and UK Independence campaigns in support of BREXIT in the United Kingdom and his is just the tip of the iceberg.

This is the level of gross manipulation of data Apple is taking a stand against.

“The fact is that an interconnected eco-system of companies and data-brokers; of purveyors of fake news and peddlers of division; of trackers and hucksters just trying to make a quick buck, is more present in our lives than it has ever been.” — Tim Cook on Privacy, 2021

What we have here are two titans of industry with massive amounts of influence and responsibility at war.

On one hand, you have Facebook who has time and time again been grilled in public forums for data harvesting of their 2.6 billion monthly active users, shadow profiles (data collected on non-Facebook users), and social media bias, and then, on the other hand, you have Apple, who have 1.5 billion active devices running iOS across iPhone and iPad, all of which are ‘tools’ that demand attention with constant notifications and habit forming user experience design.

Apple has been scrutinised in the past for its App Store policy and are currently fighting an anti-trust lawsuit filed by Epic Games over the removal of Fortnite from the App Store for violating its policies on in-app purchases. Facebook stated in December of 2020, that the company will support Epic Games’ case and is also now reportedly readying an antitrust lawsuit of its own against Apple for forcing third-party developers to follow rules that first-party apps don’t have to follow.

Zuckerberg stated in the earnings call that “Apple has every incentive to use their dominant platform position to interfere with how our apps and other apps work, which they regularly do to preference their own. And this impacts the growth of millions of businesses around the world.” and “we believe Apple is behaving anti-competitively by using their control of the App Store to benefit their bottom line at the expense of app developers and small businesses”. This is an attempt by Zuckerberg to show that Apple is using their control of the App Store to stifle the growth of small businesses but our right to know how our own data is being used should stand paramount, even if its at the expense of business growth.

Apple’s position on privacy protection ‘for the people’ and introduction of privacy ‘nutrition labelling’ is not one that just benefits users, but is one that benefits and upholds trust in the company and its products. The choices the company makes in its industries tend to form and dictate how and where the market will go. You only have to look at its previous trends in product and packaging design to see what argument I’m trying to make.

With growing concern and mainstream awareness of data use, privacy is now at the forefront of consumer trends. Just look at the emergence of VPN companies in the last couple of years. Apple’s stance on giving privacy back to the user could set a new trend into motion across the industry and usher in an age of privacy-first design.

Author: Morgan Fox

Source: Medium

Building Your Data Structure: FAIR Data

Obtaining access to the right data is a first, essential step in any Data Science endeavour. But what makes the data “right”?

The difference in datasets

Every dataset can be different, not only in terms of content, but in how the data is collected, structured and displayed. For example, how national image archives store and annotate their data is not necessarily how meteorologists store their weather data, nor how forensic experts store information on potential suspects. The problem occurs when researchers from one field need to use a dataset from a different field. The disparity in datasets is not conducive to the re-use of (multiple) datasets in new contexts.

The FAIR data principles provide guidelines to improve the Findability, Accessibility, Interoperability, and Reuse of digital assets. The emphasis is placed on the ability of computational systems to find, access, interoperate, and reuse data with no or minimal human intervention. Launched at a Lorentz workshop in Leiden in 2014, the principles quickly became endorsed and adopted by a broad range of stakeholders (e.g. European Commission, G7, G20) and have been cited widely since their publication in 2016 [1]. The FAIR principles are agnostic of any specific technological implementation, which has contributed to their broad adoption and endorsement.

Why do we need datasets that can be used in new contexts?

Ensuring that data sources can be (re)used in many different contexts can lead to unexpected results. For example, combining mental depression data with weather data can establish a correlation between mental states and weather conditions. The original data resources were not created with this reuse in mind, however, applying FAIR principles to these datasets makes this analysis possible.

FAIRness in the current crisis

A pressing example of the importance of FAIR data is the current COVID-19 pandemic. Many patients worldwide have been admitted to hospitals and intensive care units. While global efforts are moving towards effective treatments and a COVID-19 vaccine, there is still an urgent need to combine all the available data. This includes information from distributed multimodal patient datasets that are stored at local hospitals in many different, and often unstructured, formats.

Learning about the disease and its stages, and which drugs may or may not be effective, requires combining many data resources, including SARS-CoV-2 genomics data, relevant scientific literature, imaging data, and various biomedical and molecular data repositories.

One of the issues that needs to be addressed is combining privacy-sensitive patient information with open viral data at the patient level, where these datasets typically reside in very different repositories (often hospital bound) without easily mappable identifiers. This underscores the need for federated and local data solutions, which lie at the heart of the FAIR principles.

Examples of concerted efforts to build an infrastructure of FAIR data to combat COVID-19 and future virus outbreaks are in the VODAN initiative [2], the COVID-19 data portal organised by the European Bioinformatics Institute and the ELIXIR network [3].

FAIR data in Amsterdam

Many scientific and commercial applications require the combination of multiple sources of data for analysis. While providing a digital infrastructure and (financial) incentives are required for data owners to share their data, we will only be able to unlock the full potential of existing data archives when we are also able to find the datasets needed and use the data within them.

The FAIR data principles allow us to better describe individual datasets and allow easier re-use in many diverse applications beyond the sciences for which they were originally developed. Amsterdam provides fertile ground for finding partners with appropriate expertise for developing both digital and hardware infrastructures.

Author: Jaap Heringa

Source: Amsterdam Data Science

Ruim 3 jaar geleden verzorgden we de eerste leergang Business Data Scientist. Getriggerd door de vele sexy vacature teksten vroegen we ons als docenten af wat een data scientist nu exact tot data scientist maakt? In de vacatureteksten viel ons naast een enorme variëteit ook een waslijst aan noodzakelijke competenties op. De associatie met het (meestal) denkbeeldige schaap met de vijf poten was snel gelegd. Daarnaast sprak uit die vacatureteksten in 2014 vooral hoop en ambitie. Bedrijven met hoge verwachtingen op zoek naar deskundig personeel om de alsmaar groter wordende stroom data te raffineren tot waarde voor de onderneming. Wat komt daar allemaal bij kijken?

Een aantal jaar en 7 leergangen later is er veel veranderd. Maar eigenlijk ook weer weinig. De verwachtingen van bedrijven zijn nog steeds torenhoog. De data scientist komt voor in alle vormen en gedaanten. Dat lijkt geaccepteerd. Maar de kern: hoe data tot waarde te brengen en wat daarbij komt kijken blijft onderbelicht. De relevantie voor een opleiding Business Data Scientist is dus onveranderd. En eigenlijk groter geworden. De investeringen in data science zijn door veel bedrijven gedaan. Het wordt tijd om te oogsten.

Om data tot waarde te kunnen brengen is ‘verbinding’ noodzakelijk. Verbinding tussen de hard core data scientists die data als olie kunnen opboren, raffineren tot informatie en het volgens specificaties kunnen opleveren aan de ene kant. En de business mensen met hun uitdagingen aan de andere kant. In onze leergangen hebben we veel verhalen gehoord van mooie dataprojecten die paarlen voor de zwijnen bleken vanwege onvoldoende verbinding. Hoe belangrijk ook, zonder die verbinding overleeft de data scientist niet. De relevantie van een leergang Business Data Scientist is dus onveranderd. Moet iedere data scientist deze volgen? Bestaat er een functie business data scientist? Beide vragen kunnen volmondig met néé beantwoord worden. Wil je echter op het raakvlak van toepassing en data science opereren dan zit je bij deze leergang precies goed. En dat raakvlak zal meer en meer centraal gaan staan in data intensieve organisaties.

De business data scientist is iemand die als geen ander weet dat de waarde van data zit in het uiteindelijk gebruik. Vanuit dat eenvoudig uitgangspunt definieert, begeleidt, stuurt hij/zij data projecten in organisaties. Hij denkt mee over de structurele verankering van het gebruik van data science in de operationele en beleidsmatige processen van organisatie en komt met inrichtingsvoorstellen. De business data scientist kent de data science gereedschapskist door en door zonder ieder daarin aanwezige instrument ook daadwerkelijk zelf te kunnen gebruiken. Hij of zij weet echter welk stukje techniek voor welk type probleem moet worden ingezet. En omgekeerd is hij of zij in staat bedrijfsproblemen te typeren en classificeren zodanig dat de juiste technologieën en expertises kunnen worden geselecteerd. De business data scientist begrijpt informatieprocessen, kent de tool box van data science en weet zich handig te bewegen in het domein van de belangen die altijd met projecten zijn gemoeid.

De BDS leergang is relevant voor productmanagers en marketeers die data intensiever willen gaan werken, voor hard core data scientists die de verbinding willen leggen met de toepassing in hun organisatie en voor (project)managers die verantwoordelijk zijn voor het functioneren van data scientists.

De leergang BDS 2.0 wordt gekenmerkt door een actie gerichte manier van leren. Gebaseerd op een theoretisch framework dat tot doel heeft om naar de tool box van data science te kijken vanuit het oogpunt van business value staan cases centraal. In die cases worden alle fasen van het tot waarde brengen van data belicht. Van de projectdefinitie via de data analyse en de business analytics naar het daadwerkelijk gebruik. En voor alle relevante fasen leveren specialisten een deep dive. Ben je geïnteresseerd in de leergang. Download dan hier de brochure. http://www.ru.nl/rma/leergangen/bds/

Egbert Philips  

Docent BDS leergang Radboud Management Academy

Director Hammer, market intelligence   www.Hammer-intel.com

Analyst and consulting firm, Business Application Research Centre (BARC), has come out with the top BI trends based on a survey carried out on 2800 BI professionals. Compared to last year, there were no significant changes in the ranking of the importance of BI trends, indicating that no major market shifts or disruptions are expected to impact this sector.

• Data discovery and visualization,
• Self-service BI and
• Data quality and master data management
• Data labs and data science, cloud BI and data as a product were the least important trends for 2017.

• “With areas such as accounting and finance more or less under control, companies are moving to other areas of the enterprise, in particular to gain a better understanding of customer, market and competitive dynamics,” said Carsten Bange.
• Many BI trends in the past, have become critical BI components in the present.
• Many organizations were also considering trends like collaboration and sensor data analysis as critical BI components. About 20% respondents were already using BI trends like collaboration and spatial/location analysis.
• About 12% were using cloud BI and more were planning to employ it in the future. IBM's Watson and Salesforce's Einstein are gearing to meet this growth.
• Only 10% of the respondents used social media analysis.
• Sensor data analysis is also growing driven by the huge volumes of data generated by the millions of IoT devices being used by telecom, utilities and transportation industries. According to the survey, in 2017, the transport and telecoms industries would lead the leveraging of sensor data.

Chatbots, big data and the future of customer service

The rise and development of big data has paved the way for an incredible array of chatbots in customer service. Here's what to know.

Big data is changing the direction of customer service. Machine learning tools have led to the development of chatbots. They rely on big data to better serve customers.

How are chatbots changing the future of the customer service industry and what role does big data play in managing them?

Big data Leads to the deployment of more sophisticated chatbots

BI-kring published an article about the use of chatbots in HR about a month ago. This article goes deeper into the role of big data when discussing chatbots.

The following terms are more popular than ever: 'chatbot', 'automated customer service', 'virtual advisor'. Some know more, others less about process automation. One thing is for sure: if you want to sell more on the internet, handle more customers, save on personnel costs, you certainly need a chatbot. A chatbot is a conversational system that was created to stimulate intelligent conversation between a human and an automaton.

Chatbots rely on machine learning and other sophisticated data technology. They are constantly collecting new data from their interactions with customers to offer a better experience.

But how commonly used are chatbots? An estimated 67% of consumers around the world have communicated with one. That figure is going to rise sharply in the near future. In 2020, over 85% of all customer service interactions will involve chatbots.

A chatbot makes it possible to automate customer service in various communication channels, for example on a website, chat, in social media or via SMS. In practice, a customer does not have to wait for hours to receive a reply from the customer service department, a bot will provide an answer within a few seconds.

According to requirements, a chatbot may assume the role of a virtual advisor or assistant. For questions where a real person has to become involved, in analyzing the received enquiries bots can not only identify what issue the given customer is addressing but also to automatically send it to the correct person or department. Machine learning tools make it easier to determine when a human advisor is needed.

Bots supported by associative memory algorithms understand the entire content even if the interlocutor made a mistake or a typo. Machine learning makes it easier for them to decipher contextual meanings by interpreting these mistakes.

Response speed and 24/7 assistance are very important when it comes to customer service, as late afternoons and evenings are times of day when online shops experience increased traffic. If a customer cannot obtain information about a given product right there and then, it is possible that they will just abandon their basket and not come shop at that store again. Any business would want to prevent that a customer journey towards their product takes a turn the other way, especially if it's due to a lack of appropriate support.

Online store operators, trying to stay a step ahead of the competition, often decide to implement a state-of-the-art solution, which makes the store significantly more attractive and provides a number of new opportunities delivered by chatbots. Often, following the application of such a solution, website visits increase significantly. This translates into more sales of products or services.

We are not only seeing increased interest in the e-commerce industry, chatbots are successfully used in the banking industry as well. Bank Handlowy and Credit Agricole use bots to handle loyalty programmes or as assistants when paying bills.

What else can a chatbot do?

Big data has made it easier for chatbots to function. Here are some of the benefits that they offer:

• Send reminders of upcoming payment deadlines.
• Send account balance information.
• Pass on important information and announcements from the bank.
• Offer personalised products and services.
• Bots are also increasingly more often used to interact with customers wishing to order meals, taxis, book tickets, accommodation, select holiday packages at travel agents, etc.

The insurance industry is yet another area where chatbots are very useful. Since insurance companies are already investing heavily in big data and machine learning to handle actuarial analyses, it is easy for them to extend their knowledge of data technology to chatbots.

The use of Facebook Messenger chatbots during staff recruitment may be surprising for many people.

Chatbots are frequently used in the health service as well, helping to find the right facilities, arrange a visit, select the correct doctor and also find opinions about them or simply provide information on given drugs or supplements.

As today every young person uses a smartphone, social media and messaging platforms for a whole range of everyday tasks like shopping, acquiring information, sorting out official matters, paying bills etc., the use of chatbots is slowly becoming synonymous with contemporary and professional customer service. A service available 24/7, often geared to satisfy given needs and preferences.

Have you always dreamed of employees who do not get sick, do not take vacations and do not sleep? Try using a chatbot.

Big data has led to fantastic developments with chatbots

Big data is continually changing the direction of customer service. Chatbots rely heavily on the technology behind big data. New advances in machine learning and other data technology should lead to even more useful chatbots in the future.

Author: Ryan Kh

Source: SmartDataCollective

Cognitive diversity to strengthen your team

Many miles of copy have been written on the advantages of diverse teams. But all too often this thinking is only skin deep. That is it focusses on racial, gender & sexual orientation diversity.

There can be a lot more benefit in having team ;members who actually think differently. This is what is called cognitive diversity. I’ve seen that in both the teams I’ve lead and at my clients’ offices. So, when blogger Harry Powell approached me with his latestbook review, I was sold.

Harry is Director of Data Analytics at Jaguar Land Rover. He has blogged previously on th Productivity Puzzle and an Alan Turing lecture, amongst other topics. So, over to Harry to share what he has learnt about the importance of this type of diversity.

Reading about Rebel Ideas

I have just finished reading 'Rebel Ideas' by Matthew Syed. It’s not a long book, and hardly highbrow (anecdotes about 9/11 and climbing Everest, you know the kind of thing) but it made me think a lot about my team and my company.

It’s a book about cognitive diversity in teams. To be clear that’s not the same thing as demographic diversity, which is about making sure that your team is representative of the population from which it is drawn. It’s about how the people in your team think.

Syed’s basic point is that if you build a team of people who share similar perspectives and approaches the best possible result will be limited by the capability of the brightest person. This is because any diversity of thought that exists will essentially overlap. Everyone will think the same way.

But if your team comprises people who approach problems differently, there is a good chance that your final result will incorporate the best bits of everyone’s ideas, so the worst possible result will be that of the brightest person, and will it normally end up being a lot better. This is because the ideas will overlap less, and so complement each other (see note below).

Reflections on why this is a good idea

In theory, I agree with this idea. Here are a few reflections:

• The implication is that it might be better to recruit people with diverse perspectives and social skills than to simply look for the best and brightest. Obviously bright, diverse and social is the ideal.
• Often a lack of diversity will not manifest itself so much in the solutions to the questions posed, but in the selection or framing of the problems themselves.
• Committees of like-minded people not only water down ideas, they create the illusion of a limited set of feasible set of problems and solutions, which is likely to reduce the confidence of lateral thinkers to speak up.
• Strong hierarchies and imperious personalities can be very effective in driving efficient responses to simple situations. But when problems are complex and multi-dimensional, these personalities can force through simplistic solutions with disastrous results.
• Often innovation is driven not simply by the lone genius who comes up with a whole new idea, but by combining existing technologies in new ways. These new 'recombinant' ideas come together when teams are connected to disparate sets of ideas.

All this, points towards the benefits of having teams made up of people who think differently about the world. But it poses other questions.

Context guides the diversity you need

What kinds of diversity are pertinent to a given situation?

For example, if you are designing consumer goods, say mobile phones, you probably want a cross-section of ages and gender, given that different ages and genders may use those phones differently: My kids want to use games apps, but I just want email. My wife has smaller hands than me, etc.

But what about other dimensions like race, or sexual preference? Are those dimensions important when designing a phone? You would have thought that the dimension of diversity you need may relate to the problem you are trying to solve.

On the other hand, it seems that the most important point of cognitive diversity is that it makes the whole team aware of their own bounded perspectives, that there may be questions that remain to be asked, even if the demographic makeup of your team does not necessarily span wide enough to both pose and solve issues (that’s what market research is for).

So, perhaps it doesn’t strictly matter if your team’s diversity is related to the problem space. Just a mixture of approaches can be valuable in itself.

How can you identify cognitive diversity?

Thinking differently is harder to observe than demographic diversity. Is it possible to select for the former without resorting to selecting on the latter?

Often processes to ensure demographic diversity, such as standardised tests and scorecards in recruitment processes, promote conformity of thought and work against cognitive diversity. And processes to measure cognitive diversity directly (such as aptitude tests) are more contextual than are commonly admitted and may stifle a broader equality agenda.

In other words, is it possible to advance both cognitive and demographic diversity with the same process?

Even if you could identif different thinkers, what proportion of cognitive diversity can you tolerate in an organisation that needs to get things done?

I guess the answer is the proportion of your business that is complex and uncertain, although a key trait of non-diverse businesses is that their self-assessment of their need for new ideas will be limited by their own lack of perspective. And how can you reward divergent thinkers?

Much of what they do may be seen as disruptive and unproductive. Your most obviously productive people may be your least original, but they get things done.

What do I do in my team?

For data scientists, you need to test a number of skills at interview. They need to be able to think about a business problem, they need to understand mathematical methodologies, and they need to be able to code. There’s not a lot of time left for assessing originality or diversity of thought.

So what I do is make the questions slightly open-ended, maybe a bit unconventional, certainly without an obviously correct answer.

I expect them to get the questions a bit wrong. And then I see how they respond to interventions. Whether they take those ideas and play with them, see if they can use them to solve the problem. It’s not quite the same as seeking out diversity, but it does identify people who can co-exist with different thinkers: people who are open to new ways of thinking and try to respond positively.

And then try to keep a quota for oddballs. You can only have a few of them, and they’ll drive you nuts, but you’ll never regret it.

EndNote: the statistical appeal of Rebel Ideas

Note: This idea appeals to me because it has a nice machine learning analogue to it. In a regression you want your information sets to be different, ideally orthogonal. If your data is collinear, you may as well have just one regressor.

Equally, ensembles of low performing but different models often give better results than a single high-performing model.

Author: Paul Laughlin

Source: Datafloq

Connection between human and artificial intelligence moving closer to realization

What was once the stuff of science fiction is now science fact, and that’s a good thing. It is heartening to hear how personal augmentation with robotics is changing people’s lives.

A paralyzed man in France is now using a brain-controlled robotic suit to walk. The connection between brain and machine is now possible through ultra-high-speed computing power combined with deep engineering to enable a highly connected device.

Artificial Intelligence is in everyday life

We are seeing the rise of artificial intelligence in every walk of life, moving beyond the black box and being part of human life in its everyday settings.

Another example is the advent of the digital umpire in major league baseball. The angry disputes of players and fans challenging the umpire and holding of breaths for the replay may become a thing of the past with deep precision of instant decisions from an unbiased, non-impassioned, and non-human ump.

Augmented reality and virtual reality are also becoming a must for business. They have moved into every aspect from medicine to mining across design, manufacturing, logistics and service and are a familiar business tool delivering multi-dimensional immediate insights that were previously hard to find.

For example, people are using digital twin technology to see deeply into equipment, wherever it is, and diagnose and fix problems, or to take a global view of business operations through a digital board room.

What’s changed? Fail fast, succeed sooner!

Every technology takes time to find its groove as early adopters experiment and find mass uses for it. There is a usual cycle of experimentation, fast failure is a necessary part of discovering best applications for technology. We all saw Google Goggles fail to find market traction, but in its current generation, it is an invaluable addition to provide valuable information to people in the field, repairing equipment and needed expertise on site.

Speed, Intelligence, and Connection make it happen

The Six Million Dollar Man for business should be able to connect to the brain, providing instant feedback to the operations of the business based on actual experience in the field. It has to operate in the speed of a heartbeat and use predictive technologies. (Nerd Alert: Speaking of the Six Million Dollar Man, it should come as no surprise that the titular character has been upgraded to 'The Six Billion Dollar Man' in the upcoming movie starring Mark Walberg.)

Think of all the stuff our brain is doing even as we walk, balancing our bodies as we are in motion, making adjustments as we turn our head or land our feet. Predicting where our body will be so that the weight of our limbs can be adjusted, the brain needs instant feedback from all our senses to make decisions in realtime that appear to be 'natural'.

Business, too, needs systems that are deeply connected, predictive, and high speed, balancing the desire for movement to optimizing the operations to make it happen. That requires a new architecture that is lightning fast using memory rather than disk processing, using artificial intelligence to optimize decisions that are too fast to make on our own, to keep a pulse on the business and to predict with machine learning.

The fundamental architecture is different. It has to work together and be complete; it is no good having leg movements from one vendor and head movements from another. In a world where speed and sensing has to cover the whole body, it needs to work in unison.

We can’t wait to see how these new architectures will change the world.

Author: David Sweetman

Source: Dataversity

Context & Uncertainty in Web Analytics

Trying to make decisions with data

“If a measurement matters at all, it is because it must have some conceivable effect on decisions and behaviour. If we can’t identify a decision that could be affected by a proposed measurement and how it could change those decisions, then the measurement simply has no value” - Douglas W. Hubbard, How to Measure Anything: Finding the Value of Intangibles in Business, 2007

Like many digital businesses we use web analytics tools that measure how visitors interact with our websites and apps. These tools provide dozens of simple metrics, but in our experience their value for informing a decision is close to zero without first applying a significant amount of time, effort and experience to interpret them.

Ideally we would like to use web analytics data to make inferences about what stories our readers value and care about. We can then use this to inform a range of decisions: what stories to commission, how many articles to publish, how to spot clickbait, which headlines to change, which articles to reposition on the page, and so on.

Finding what is newsworthy can not and should not be as mechanistic as analysing an e-commerce store, where the connection between the metrics and what you are interested in measuring (visitors and purchases) is more direct. We know that — at best — this type of data can only weakly approximate what readers really think, and too much reliance on data for making decisions will have predictable negative consequences. However, if there is something of value the data has to say, we would like to hear it.

Unfortunately, simple web analytics metrics fail to account for key bits of context that are vital if we want to understand if their values are higher or lower than what we should expect (and therefore interesting).

Moreover, there is inherent uncertainty in the data we are using, and even if we can tell whether the value is higher or lower than expected, it is difficult to tell whether this is just down to chance.

Good analysts, familiar with their domain often get good at doing the mental gymnastics required to account for context and uncertainty, so they can derive the insights that support good decisions. But doing this systematically when presented with a sea of metrics is rarely possible or the best use of an analyst’s valuable sense-making skills. Rather than all their time being spent trying to identify what is unusual, it would be better if their skills could be applied to learning why something is unusual or deciding how we might improve things. But if all of our attention is focused on the lower level what questions, we never get to the why or how questions — which is where we stand a chance of getting some value from the data.

Context

“The value of a fact shrinks enormously without context” - Howard Wainer, Visual Revelations: Graphical Tales of Fate and Deception from Napoleon Bonaparte to Ross Perot, 1997

Take two metrics that we would expect to be useful — how many people start reading an article (we call this readers), and how long they spend on it (we call this the average dwell time). If the metrics worked as intended, they could help us identify the stories our readers care about, but in their raw form, they tell us very little about this.

• Readers: If an article is in a more prominent position on the website or app, more people will see it and click on it.
• Dwell time: If an article is longer, on average, people will tend to spend more time reading it.

Counting the number of readers tells us more about where an article was placed, and dwell time more about the length of the article than anything meaningful.

It’s not just length and position that matter. Other context such as the section, the day of the week, how long since it was published, and whether people are reading it on our website or apps all systematically influence these numbers. So much so, that we can do a reasonable job of predicting how many readers an article will get and how long they will spend on it by only looking at its context, and completely ignoring the content of the article.

From this perspective, articles are a victim of circumstance, and the raw metrics we see in so many dashboards tell us more about their circumstances than anything more meaningful — it’s all noise and very little signal.

Knowing this, what we really want to understand is how much better or worse an article did than we would expect, given that context. In our newsroom, we do this by turning each metric (readers, dwell time and some others) into an index that compares the actual metric for an article to it’s expected value. We score it on a scale from 1 to 5, where 3 is expected, 4 or 5 is better than expected and 1 or 2 is worse than expected.

Article A: a longer article in a more prominent position. Neither the number of readers nor the time they spent reading it was different from what we would expect (both indices = 3).

Article B: a shorter article in a less prominent position. Whilst it had the expected number of readers (index = 3), they spent longer reading it than we would expect (index = 4).

The figures above show how we present this information when looking at individual articles. Article A had 7,129 readers, more than four thousand more readers than article B, and people spent 2m 44s reading article A, almost a minute longer than article B. A simple web analytics display would pick article A as the winner on both counts by a large margin. And completely mislead us.

Once we take into account the context, and calculate the indices, we find that both articles had about as many readers as we would expect, no more or less. Even though article B had four thousand fewer, it was in a less prominent position, and so we wouldn’t expect so many. However, people did spend longer reading article B than we would expect, given factors such as it’s length (it was shorter than article A).

The indices are the output of a predictive model, which predicts a certain value (e.g. number of readers), based on the context (the features in the model). The difference between the actual value and the predicted value (the residuals in the model) then form the basis of the index, which we rescale into the 1–5 score. An additional benefit is that we also have a common scale for different measures, and a common language for discussing these metrics across the newsroom.

Unless we account for context, we can only really use data for inspection: ‘Just tell me which article got me the most readers, I don’t care why’. If the article only had more readers because it was at the top of the edition we’re not learning anything useful from the data, and at worst it creates a self fulfilling feedback loop (more prominent articles get more readers — similar to the popularity bias that can occur in recommendation engines).

In his excellent book Upstream, Dan Heath talks about moving from data for inspection to data for learning. Data for learning is fundamental if we want to make better decisions. If we want to use data for learning in the newsroom, it’s incredibly useful to be able to identify which articles are performing better or worse than we would expect, but that is only ever the start. The real learning comes from what we do with that information, trying something different, and seeing if it has a positive effect on our readers’ experience.

“Using data for inspection is so common that leaders are sometimes oblivious to any other model.” - Dan Heath, Upstream: The Quest to Solve Problems Before They Happen, 2020

Uncertainty

“What is not surrounded by uncertainty cannot be truth” - Richard Feynman (probably)

The metrics presented in web analytics tools are incredibly precise. 7,129 people read the article we looked at earlier. How do we compare that to an article with 7,130 readers? What about one with 8,000? When presented with numbers, we can’t help making comparisons, even if we have no idea whether the difference matters.

We developed our indices to avoid meaningless comparisons that didn’t take into account context, but earlier versions of our indices were displayed in a way that suggested more preciseness than they provided — we used a scale from 0 to 200 (with 100* as expected).

*Originally we had 0 as our expected value, but quickly learnt that nobody likes having a negative score for their article, but something below 100 is more palatable.

Predictably, people started worrying about small differences in the index values between articles. ‘This article scored 92 , but that one scored 103, that second article did better, let’s look at what we can learn from it’. Sadly the model we use to generate the index is not that accurate, and models, like data have uncertainty associated with them. Just as people agonise over small meaningless differences in raw numbers, the same was happening with the indices, and so we moved to a simple 5 point scale.

Most articles get a 3, which can be interpreted as ‘we don’t think there is anything to see here, the article is doing as well as we’d expect on this measure’. An index of 2 or 1 means it is doing a bit worse or a lot worse than expected, and a 4 or a 5 means it is doing a bit better or a lot better than expected.

In this format, the indices provide just enough information for us to know — at a glance — how an article is doing. We use this alongside other data visualisations of indices or raw metrics where more precision is helpful, but in all cases our aim is to help focus attention on what matters, and free up time to validate these insights and decide what to do with them.

Why are context and uncertainty so often ignored?

These problems are not new and covered in many great books on data sense-making — some are decades old, but more recently Howard Wainer, Stephen Few and R J Andrews.

Practical guidance on dealing with uncertainty is easier to come by, but in our experience, thinking about context is trickier. From some perspectives this is odd. Predictive models — the bread and butter of data scientists — inherently deal with context as well as uncertainty, as do many of the tools for analysing time series data and detecting anomalies (such as statistical process control). But we are also taught to be cautious when making comparisons where there are fundamental differences between the things we are measuring. Since there are so many differences between the articles we publish, from length, position, who wrote them, what they are about, to the section and day of week on which they appear, we are left wondering whether we can or should use data to compare any of them. Perhaps the guidance on piecing all of this together to build better measurement metrics is less common, because how you deal with context is so contextual.

Even if you set out on this path, there are many mundane reasons to fail. Often the valuable context is unavailable. It took us months to bring basic metadata about our articles— such as length and the position in which they appear— into the same system as the web analytics data. An even bigger obstacle is how much time it takes just to maintain a reliable metrics system (digital products are constantly changing, and this often breaks the web analytics data, including ours as I wrote this). Ideas for improving metrics often stay as ideas or proof of concepts that are not fully rolled out as you deal with these issues.

If you do get started, there are myriad choices to make to account for context and uncertainty— from technical to ethical — all involving value judgements. If you stick with a simple metric you can avoid these choices. Bad choices can derail you, but even if you make good ones, if you can’t adequately explain what you have done, you can’t expect the people who use the metrics to trust them. By accounting for context and uncertainty you may replace a simple (but not very useful) metric with something that is in theory more useful, but the opaqueness causes more problems than it solves. Even worse, people place too much trust in the metric and use it without questioning it.

As for using data to make decisions. We will leave that for another post. But if the data is all noise and no signal, how do you present it in a clear way so the people using it understand what decisions it can help them make? The short answer is you can’t. But if the pressure is on to present some data, it is easier to passively display it in a big dashboard, filled with metrics and leave it to others to work out what to do, in the same way passive language can shield you if you have nothing interesting to say (or bullshit as Carl T. Bergstrom would call it). This is something else we have battled with, and we have tried to avoid replacing big dashboards filled with metrics with big dashboards filled with indices.

Adding an R for reliable and an E for explainable, we end up with a checklist to help us avoid bad — or CRUDE — metrics (Context Reliability Uncertainty Decision orientated Explainability). Checklists are always useful, as it’s easy to forget what matters along the way.

Anybody promising a quick and easy path to metrics that solve all your problems is probably trying to sell you something. In our experience, it takes time and a significant commitment by everybody involved to build something better. If you don’t have this, it’s tough to even get started.

Non-human metrics

Part of the joy and pain of applying these principles to metrics used for analytics — that is, numbers that are put in front of people who then use them to help them make decisions — is that it provides a visceral feedback loop when you get it wrong. If the metrics cannot be easily understood, if they don’t convey enough information (or too much), if they are biased, or if they are unreliable or if they just look plain wrong vs. everything the person using them knows, you’re in trouble. Whatever the reason, you hear about it pretty quickly, and this is a good motivator for addressing problems head on if you want to maintain trust in the system you have built.

Many metrics are not designed to be consumed by humans. The metrics that live inside automated decision systems are subject to many of the same considerations, biases and value judgements. It is sobering to consider the number of changes and improvements we have made based on the positive feedback loop from people using our metrics in the newsroom on a daily basis. This is not the case with many automated decision systems.

Author: Dan Gilbert

Source: Medium

Data has key contributions for creating a single view of the customer that can be used to improve your business and better understand those involved in the market you serve. In order to thrive in the current economic environment, businesses need to know their customers very well in order to provide exceptional customer service. To do so, they must be able to rapidly understand and react to customer shopping behaviors. To properly interpret and react to customer behaviors, businesses need a complete, single view of their customers. What does that mean? A Single View of the Customer (SVC) allows a business to analyze and visualize all the relevant information surrounding their customers, such as transactions, products, demographics, marketing, discounting, etc. Unfortunately, the IT systems that are typically found in mid-to large-scale companies have separated much of the relevant customer data into individual systems. Marketing information, transaction data, website analytics, customer profiles, shipping information, product information, etc. are often kept in different data repositories. This makes the implementation of SVC potentially challenging. First of all, let’s examine two scenarios where a company’s ability to fuse all these external sources into an SVC can provide tremendous value. Afterwards, we will pay attention to strategies for implementing an SVC.

Call center

Customer satisfaction has a major impact on the bottom line of any business. Studies found that 78 percent of customers have failed to complete a transaction because of poor customer service. A company’s call center is a key part ofmaintaining customer satisfaction. Customers interacting with the call center typically do so because there is already a problem – a missing order, a damaged or incorrect product. It is critical that the call center employee can resolve the problem as quickly as possible. Unfortunately, due to the typical IT infrastructure discussed above, the customer service representative often faces a significant challenge. For example, in order to locate the customer’s order, access shipping information and the potential to find a replacement product to ship, the customer service representative may have to log in to three or more different systems (often, the number is significantly higher). Every login to a new system increases the time the customer has to wait, decreasing their satisfaction, and every additional system adds to the probability of diturbances or even failure.

Sales performance

Naturally, in order to maximize revenue, it is critical to understand numerous key metrics including, but not limited to:

• What products are/are not doing well?
• What products are being purchased by your key customer demographic groups?
• What items are being purchased together (basket analysis)?
• Are there stores with inventory problems (overstocked/understocked)?

Once again, the plethora of data storage systems poses a challenge. To gather the data required to perform the necessary analytics, numerous systems will have to be queried. As with the call center scenario, this is often performed manually, via “swivel-chair integration.” This means that an analyst will have to manually login to each system, execute a query to get the necessary data, store that data in a temporary location (often in Microsoft Excel™), and then repeat that process for each independent data store. Finally, once the data is gathered the process of performing the actual analysis can begin. The process of gathering the necessary data often takes longer than the actual analysis. Even in medium-sized companies this process can involve numerous people to get the analysis done as quickly as possible. Still, the manual nature of this process means that it is not only expensive to perform (in terms of resources), but it occurs at a much slower pace than ideal to rapidly make business decisions. The fact that performing even the most basic and critical analytics is so expensive and time consuming often prevents companies from taking the next steps, even though those steps could turn out critical to the business' sales. One of those potential next steps is moving the full picture of the customer directly into the stores. When sales associates can immediately access customer information, they are able to provide a more personalized customer experience, which is likely to increase customer satisfaction and average revenue per sale. Another opportunity where the company can see tremendous sales impact is in moving from reactive analytics to predictive analytics. When a company runs traditional retail metrics – as previously described – they are typically either done as part of a regular reporting cycle or in response to an event, such as a sale, in order to understand the impact of that event on the business. While no one is likely to dispute the value of those analytics, the fact is that the company is now merely reacting to events that have already happened. We can try to use some sort of advanced analytic methods to predict how our customers may behave in the future based on their past actions, but as we so often hear from financial analysts, past performance is not indicative of future results. However, if we can take our SVC, that links together all of their past actions, and tie in information about what the customer intends to do in the future (like Prosper Insight Analytics', we now have a roadmap of customer intent that we can use to make key business decisions.

Creating and implementing a Single View of the Customer

To be effective, an implementation of SVC must be driven from a unified data fabric. Attempting to create an SVC by directly connecting the application to each of the necessary data sources will be extremely time consuming and will prove highly challenging to implement. A data fabric can connect the necessary data to provide an operational analytic environment upon which to base the SVC. The data fabric driving the SVC should meet the following requirements:

• It must connect all the relevant data from the existing data sources in a central data store. This provides the ability to perform the complex analytics across all the linked information.
• It needs to be easily modified to support new data. As the business grows and evolves, learning to leverage its new SVC, additional data sources will typically be identified as being helpful. These should be easy to integrate into the system.
• The initial implementation must be rapid. Ideally, a no-code solution should be implemented. Companies rarely have the resources to support expensive, multi-month IT efforts.
• It should not disrupt existing systems. The data fabric should provide an augmented data layer that supports the complex analytic queries required by the SVC without disrupting the day-to-day functionality of the existing data stores or applications.

Conclusion

A well-built SVC can have a significant, positive impact on a company’s bottom line. Customer satisfaction can be increased and analysis can move from being reactive to being predictive of customer behavior. This effort will likely require that a data fabric be developed to support the application, but new technologies now make it possible to rapidly create that fabric using no-code solutions, thereby making feasible the deployment of a Single View of the Customer.

Author: Clark Richey

Source: Smart Data Collective

Dashboard storytelling: The perfect presentation (part 1)

Plato famously said that “those who tell stories rule society.” This statement is as true today as it was in ancient Greece, perhaps even more so in modern times.

In the contemporary world of business, the age-old art of storytelling is far from forgotten: rather than speeches on the Senate floor, businesses rely on striking data visualizations to convey information, drive engagement, and persuade audiences.

By combining the art of storytelling with the technological capabilities of dashboard software, it’s possible to develop powerful, meaningful, data-backed presentations that not only move people but also inspire them to take action or make informed, data-driven decisions that will benefit your business.

As far back as anyone can remember, narratives have helped us make sense of the sometimes complicated world around us. Rather than just listing facts, figures, and statistics, people used gripping, imaginative timelines, bestowing raw data with real context and interpretation. In turn, this got the attention of listeners, immersing them in the narrative, thereby offering a platform to absorb a series of events in their mind’s eye precisely the way they unfolded.

Here we explore data-driven, live dashboard storytelling in depth, looking at storytelling with KPIs and the dynamics of a data storytelling presentation while offering real-world storytelling presentation examples.

First, we’ll delve into the power of data storytelling as well as the general dynamics of a storytelling dashboard and what you can do with your data to deliver a great story to your audience. Moreover, we will offer dashboard storytelling tips and tricks that will help you make your data-driven narrative-building efforts as potent as possible, driving your business into exciting new dimensions. But let’s start with a simple definition.

“You’re never going to kill storytelling, because it’s built in the human plan. We come with it.” – Margaret Atwood

What is dashboard storytelling?

Dashboard storytelling is the process of presenting data in effective visualizations that depict the whole narrative of key performance indicators, business strategies and processes in the form of an interactive dashboard on a single screen, and in real-time. Storytelling is indeed a powerful force, and in the age of information, it’s possible to use the wealth of insights available at your fingertips to communicate your message in a way that is more powerful than you could ever have imagined. So, let's take a look at the top tips and tricks to be able to successfully create your own story with a few clicks.

4 Tricks to get started with dashboard storytelling

Big data commands big stories.

Forward-thinking business people turn to online data analysis and data visualizations to display colossal volumes of content in a few well-designed charts. But these condensed business insights may remain hidden if they aren’t communicated with words in a way that is effective and rewarding to follow. Without language, business people often fail to push their message through to their audience, and as such, fail to make any real impact.

Marketers, salespeople, and entrepreneurs are today’s storytellers. They are wholly responsible for their data story. People in these roles are often the bridge between their data and the forum of decision-makers they’re looking to encourage to take the desired action.

Effective dashboard storytelling with data in a business context must be focused on tailoring the timeline to the audience and choosing one of the right data visualization types to complement or even enhance the narrative.

To demonstrate this notion, let’s look at some practical tips on how to prepare the best story to accompany your data.

1. Start with data visualization

This may sound repetitive, but when it comes to a dashboard presentation, or dashboard storytelling presentation, it will form the foundation of your success: you must choose your visualization carefully.

Different views answer different questions, so it’s vital to take care when choosing how to visualize your story. To help you in this regard, you will need a robust data visualization tool. These intuitive aids in dashboard storytelling are now ubiquitous and provide a wide array of options to choose from, including line charts, bar charts, maps, scatter plots, spider webs, and many more. Such interactive tools are rightly recognized as a more comprehensive option than PowerPoint presentations or endless Excel files.

These tools help both in exploring the data and visualizing it, enabling you to communicate key insights in a persuasive fashion that results in buy-in from your audience.

But for optimum effectiveness, we still need more than a computer algorithm.. Here we need a human to present the data in a way that will make it meaningful and valuable. Moreover, this person doesn’t need to be a common presenter or a teacher-like figure. According to research carried out by Stanford University, there are two types of storytelling: author- and reader-driven storytelling.

An author-driven narrative is static and authoritative because it dictates the analysis process to the reader or listener. It’s like analyzing a chart printed in a newspaper. On the other hand, reader-driven storytelling allows the audience to structure the analysis on their own. Here, the audience can choose the data visualizations that they deem meaningful and interact with them on their own by drilling down to more details or choosing from various KPI examples they want to see visualized. They can reach out for insights that are crucial to them and make sense out of data independently. A different story may need a different type of stoeytelling.

2. Put your audience first

Storytelling for a dashboard presentation should always begin with stating your purpose. What is the main takeaway from your data story? It should be clear that your purpose is to motivate the audience to take a certain action.

Instead of thinking about your business goals, try to envision what your listeners are seeking. Each member of your audience, be that a potential customer, future business partner, or stakeholder, has come to listen to your data storytelling presentation to gain a profit for him or herself. To better meet your audience’s expectations and gain their trust (and money), put their goals first in the determination of the line of your story.

Needless to say, before your dashboard presentation, try to learn as much as you can about your listeners. Put yourself in their shoes: Who are they? What do they do on a daily basis? What are their needs? What value can they draw from your data for themselves?

The better you understand your audience, the more they will trust you and follow your idea.

3. Don’t fill up your data storytelling with empty words

Storytelling with data, rather than just presenting data visualizations, brings the best results. That said, there are certain enemies of your story that make it more complicated than enlightening and turn your efforts into a waste of time.

The first things that could cause some trouble are the various technology buzzwords that are devoid of any defined meaning. These words don’t create a clear picture in your listeners’ heads and are useless as a storytelling aid. In addition, to under-informing your audience, buzzwords are a sign of your lazy thinking and a herald that you don’t have anything unique or meaningful to say. Try to add clarity to your story by using more precise and descriptive narratives that truly communicate your purpose.

Another trap can be the use of your industry jargon to sound more professional. The problem here is that it may not be the jargon of your listeners’ industry, they may not comprehend your narrative. Moreover, some jargon phrases have different meanings depending on the context they are used in. They mean one thing in the business field and something else in everyday life. Generally they reduce clarity and can also convey the opposite meaning of what you intend to communicate in your data storytelling.

Don’t make your story too long, focus on explaining the meaning of data rather than the ornateness of your language, and humor of your anecdotes. Avoid overusing buzzwords or industry jargon and try to figure out what insights your listeners want to draw from the data you show them.

4. Utilize the power of storytelling

Before we continue our journey into data-powered storytelling, we’d like to further illustrate the unrivaled the power of offering your audience, staff, or partners inspiring narratives by sharing these must-know insights:

• Recent studies suggest that 80% of today’s consumers want brands to tell stories about their business or products.

• The average person processes 100 to 500 digital words every day. By taking your data and transforming it into a focused, value-driven narrative, you stand a far better chance of your message resonating with your audience and yielding the results you desire.

• Human beings absorb information 60 times faster with visuals than with linear text-based content alone. By harnessing the power of data visualization to form a narrative, you’re likely to earn an exponentially greater level of success from your internal or external presentations.

Please also take a look at part 2 of this interesting read, including presentation tips and examples of dashboard storytelling.

Author: Sandra Durcevic

Source: Datapine

Dashboard storytelling: The perfect presentation (part 2)

In the first part of this article, we have introduced the phenomenon of dashboard storytelling and some tips and tricks to get started with it. If you haven´t read part 1 of this article, make sure you do that! You can find part 1 here.

How to present a dashboard – 6 Tips for the perfect dashboard storytelling presentation

Now that we’ve covered the data-driven storytelling essentials, it’s time to dig deeper into ways that you can make maximum impact with your storytelling dashboard presentations.

Business dashboards are now driving forces for visualization in the field of business intelligence. Unlike their predecessors, a state-of-the-art dashboard builder gives presenters the ability to engage audiences with real-time data and offer a more dynamic approach to presenting data compared to the rigid, linear nature of, say, Powerpoint for example.

With the extra creative freedom data dashboards offer, the art of storytelling is making a reemergence in the boardroom. The question now is: What determines great dashboarding?

Without further ado, here are six tips that will help you to transform your presentation into a story and rule your own company through dashboard storytelling.

1. Set up your plan

Start at square one on how to present a dashboard: outline your presentation. Like all good stories, the plot should be clear, problems should be presented, and an outcome foreshadowed. You have to ask yourself the right data analysis questions when it comes to exploring the data to get insights, but you also need to ask yourself the right questions when it comes to presenting such data to a certain audience. Which information do they need to know or want to see? Make sure you have a concise storyboard when you present so you can take the audience along with you as you show off your data. Try to be purpose-driven to get the best dashboarding outcomes, but don’t entangle yourself in a rigid format that is unchangeable.

2. Don’t be afraid to show some emotion

Stephen Few, a leading design consultant, explains on his blog that “when we appeal to people’s emotions strictly to help them personally connect with information and care about it, and do so in a way that draws them into reasoned consideration of the information, not just feeling, we create a path to a brighter, saner future”. Emotions stick around much longer in a person’s psyche than facts and charts. Even the most analytical thinkers out there will be more likely to remember your presentation if you can weave elements of human life and emotion. How to present a dashboard with emotion? By adding some anecdotes, personal life experiences that everyone can relate to, or culturally shared moments and jokes.

However, do not rely just on emotions to make your point. Your conclusions and ideas need to be backed by data, science, and facts. Otherwise, and especially in business contexts, you might not be taken seriously. You’d also miss an opportunity to help people learn to make better decisions by using reason and would only tap into a “lesser-evolved” part of humanity. Instead, emotionally appeal to your audience to drive home your point.

3. Make your story accessible to people outside your sector

Combining complicated jargon, millions of data points, advanced math concepts, and making a story that people can understand is not an easy task. Opt for simplicity and clear visualizations to increase the level of audience engagement.

Your entire audience should be able to understand the points that you are driving home. Jeff Bladt, the director of Data Products Analytics at DoSomething.org, offered a pioneering case study on accessibility through data. When commenting on how he goes from 350 million data points to organizational change, he shared: “By presenting the data visually, the entire staff was able to quickly grasp and contribute to the conversation. Everyone was able to see areas of high and low engagement. That led to a big insight: Someon outside the analytics team noticed that members in Texas border towns were much more engaged than members in Northwest coastal cities.”

Making your presentation accessible to laypeople opens up more opportunities for your findings to be put to good use.

4. Create an interactive dialogue

No one likes being told what to do. Instead of preaching to your audience, enable them to be a part of the presentation througinteractive dashboard features. By using real-time data, manipulating data points in front of the audience, and encouraging questions during the presentation, you will ensure your audiences are more engaged as you empower them to explore the data on their own. At the same time, you will also provide a deeper context. The interactivity is especially interesting in dashboarding when you have a broad target audience: it onboards newcomers easily while letting the ‘experts’ dig deeper into the data for more insights.

5. Experiment

Don’t be afraid to experiment with different approaches to storytelling with data. Create a dashboard storytelling plan that allows you to experiment, test different options, and learn what will build the engagement among your listeners and make sure you fortify your data storytelling with KPIs (Key Performance Indicators). As you try and fail by making them fall asleep or check their email, you will only learn from it and get the information on how to improve your dashboarding and storytelling with data techniques, presentation after presentation.

6. Balance your words and visuals wisely

Last but certainly not least is a tip that encompasses all of the above advice but also offers a means of keeping it consistent, accessible, and impactful from start to finish balance your words and visuals wisely.

What we mean here is that in data-driven storytelling, consistency is key if you want to grip your audience and drive your message home. Our eyes and brains focus on what stands out. The best data storytellers leverage this principle by building charts and graphs with a single message that can be effortlessly understood, highlighting both visually and with words the strings of information that they want their audience to remember the most.

With this in mind, you should keep your language clear, concise, and simple from start to finish. While doing this, use the best possible visualizations to enhance each segment of your story, placing a real emphasis on any graph, chart, or sentence that you want your audience to take away with them.

Every single element of your dashboard design is essential, but by emphasizing the areas that really count, you’ll make your narrative all the more memorable, giving yourself the best possible chance of enjoying the results you deserve.

The best dashboard storytelling examples

Now that we’ve explored the ways in which you can improve your data-centric storytelling and make the most of your presentations, it’s time for some inspiring storytelling presentation examples. Let’s start with a storytelling dashboard that relates to the retail sector.

1. A retailer’s store dashboard with KPIs

The retail industry is an interesting one as it has particularly been disrupted with the advent of online retailing. Collecting data analytics is extremely important for this sector as it can take an excellent advantage out of analytics because of its data-driven nature. And as such, data storytelling with KPIs is a particularly effective method to communicate trends, discoveries and results.

The first of our storytelling presentation examples serves up the information related to customers’ behavior and helps in identifying patterns in the data collected. The specific retail KPIs tracked here are focused on the sales: by division, by items, by city, and the out-of-stock items. It lets us know what the current trends in customers’ purchasing habits are and allow us to break down this data according to a city or a gender/age for enhanced analysis. We can also anticipate any stock-out to avoid losing money and visualize the stock-out tendencies over time to spot any problems in the supply chain.

2. A hospital’s management dashboard with KPIs

This second of our data storytelling examples delivers the tale of a busy working hospital. That might sound a little fancier than it is, but it’s of paramount importance. All the more when it comes to public healthcare, a sector very new to data collection and analytics that has a lot to win from it in many ways.

For a hospital, a centralized dashboard is a great ally in the everyday management of the facility. The one we have here gives us the big picture of a complex establishment, tracking several healthcare KPIs.

From the total admissions to the total patients treated, the average waiting time in the ER, or broken down per division, the story told by the healthcare dashboard is essential. The top management of this facility have a holistic view to run the operations more easily and efficiently and can try to implement diverse measures if they see abnormal figures. For instance, an average waiting time for a certain division that is way higher than the others can shed light on some problems this division might be facing: lack of staff training, lack of equipment, understaffed unit, etc.

All this is vital for the patient’s satisfaction as well as the safety and wellness of the hospital staff that deals with life and death every day.

3. A human resources (HR) recruitment dashboard with KPIs

The third of our data storytelling examples relates to human resources. This particular storytelling dashboard focuses on one of the most essential responsibilities of any modern HR department: the recruitment of new talent.

In today’s world, digital natives are looking to work with a company that not only shares their beliefs and values but offers opportunities to learn, progress, and grow as an individual. Finding the right fit for your organization is essential if you want to improve internal engagement and reduce employee turnover.

The HR KPIs related to this storytelling dashboard are designed to enhance every aspect of the recruitment journey, helping to drive down economical efficiencies and improving the quality of hires significantly.

Here, the art of storytelling with KPIs is made easy. This HR dashboard offers a clear snapshot into important aspects of HR recruitment, including the cost per hire, recruiting conversion or success rates, and the time to fill a vacancy from initial contact to official offer.

With this most intuitive of data storytelling examples, building a valuable narrative that resonates with your audience is made easy, and as such, it’s possible to share your recruitment insights in a way that fosters real change and business growth.

Final words of advice

One of the major advantages of working with dashboards is the improvement they have made to data visualization. Don’t let this feature go to waste with your own presentations. Place emphasis on making visuals clear and appealing to get the most from your dashboarding efforts.

Transform your presentations from static, lifeless work products into compelling stories by weaving an interesting and interactive plot line into them.

If you haven't read part 1 of this article yet, you can find it here.

Author: Sandra Durcevic

Source: Datapine

Data access: the key to better decision making

When employees have better access to data, they end up making better decisions.

Companies across sectors are already well in the habit of collecting relevant historical and business data to make projections and forecast the unknown future. They’re collecting this data at such a scale that 'big data' has become a buzzword technology. They want lots of it because they want an edge wherever they can get it. Who wouldn’t?

But it’s not only the quantity and quality of the data a company collects that play a pivotal role in how that company moves forward, it’s also a question of access. When businesses democratize access to that data such that it’s accessible to workers throughout a hierarchy (and those workers end up actually interacting with it), it increases the quality of decisions made on lower rungs of the ladder. Those decisions end up being more often data-informed, and data is power.

But that’s easier said than done lately. Businesses have no issue collecting data nowadays, but they do tend to keep it cordoned off.

Data sticks to the top of a business hierarchy

A business’s C-suite (often with help from a technical data science team) makes the big-picture decisions that guide the company’s overall development. This means the employees using data to inform a chosen course of action (like last year’s revenue versus this year’s revenue, or a certain client’s most common order) are either highly ranked within the company, or are wonky data specialists. Data lives behind a velvet rope, so to speak.

But this data would be eminently useful to people throughout an organization, regardless of their rank or tenure. Such a level of access would make it more likely that data guides every decision, and that would lead to more desirable business outcomes over time. It might even overtly motivate employees by subtly reinforcing the idea that results are tracked and measured.

Data tends not to trickle down to the appropriate sources

Who better to have a clear view of the business landscape than the employees who toe the front lines every day? What would change if disparate employees scattered throughout an organization suddenly had access to actionable data points? These are the people positioned to actually make a tweak or optimization from the get-go. Whoever comes up with a data-informed strategy on a strong way forward, these are the people actually implementing it. But an organization-level awareness of an actionable data point doesn’t necessarily equate to action.

As previously established, data has a high center of gravity. It is managerial food for thought on the way to designing and executing longer-term business strategies.

But when companies change their culture around access to data and make it easy for everyone to interact with data, they make every worker think like such a strategist.

By the time a piece of data reaches an appropriate source, it’s notnecessarily in a form he or she can’t interact with or understand

As much as managers might like to think otherwise, there are people in their organization thinking in less than granular terms. They aren’t necessarily thinking about the costs their actions may or may not be having on the company, they don’t think about the overall bottom line. That’s why it’s important that data be in a form that people can use or understand, because it doesn’t always reach them that way.

Getting data into a useable, understandable form happens by preserving connection between departments and avoiding disconnects.

There seems to be a big data disconnect at the intersection of engineering and product development

This is the intersection is where a business’s technical prowess meets its ability to design a great product. While the two pursuits are clearly related to one another on the way to great product design, it’s rare that one person should excel at both.

The people who design groundbreaking machine learning algorithms aren’t necessarily the people who design a groundbreaking consumer product, and vice versa. They need each other’s help to understand each other.

But data is the shared language that makes understanding possible. Not everyone has years of data science training, not everyone has business leadership experience, but even people doing menial things can still benefit from great access to data. Coming across the year’s growth goal, for example, might trigger a needle-moving idea from someone on how to actually get there. Great things happen when employees build a shared understanding of the raw numbers that drive everything they do.

Businesses already collect so much data in the course of their day-to-day operations. But they could start using that data more effectively by bringing it out from behind the curtain, presenting employees across the board with easy access and interaction for it. The motivation for doing so should be clear: when more people think about the same problem in the same terms, that problem is more likely to be solved.

All they need is access to the data that makes it possible.

Author: Simone Di Somma

Source: Insidebigdata

Data alone is not enough, storytelling matters - part 1

Crafting meaningful narratives from data is a critical skill for all types of decision making, in business, and in our public discourse

As companies connect decision-makers with advanced analytics at all levels of their organizations, they need both professional and citizen data scientists who can extract value from that data and share. These experts help develop process-driven data workflows, ensuring employees can make predictive decisions and get the greatest possible value from their analytics technologies.

But understanding data and communicating its value to others are two different skill sets. Your team members’ ability to do the latter impacts the true value you get from your analytics investment. This can work for or against your long-term decision-making and will shape future business success.

There are between stories and their ability to guide people’s decisions, even in professional settings. Sharing data in a way that adds value to decision-making processes still requires a human touch. This is true even when that data comes in the form of insights from advanced analytics.

That’s why data storytelling is such a necessary activity. Storytellers convert complex datasets into full and meaningful narratives, rich with visualizations that help guide all types of business decisions. This can happen at all levels of the organization with the right tools, skill sets, and workflows in place. This article highlights the importance of data storytelling in enterprise organizations and illustrates the value of the narrative in decision-making processes.

What is data storytelling?

Data storytelling is an acquired skill. Employees who have mastered it can make sense out of a body of data and analytics insights, then convey their wisdom via narratives that make sense to other team members. This wisdom helps guide decision making in an honest, accurate, and valuable way.

Reporting that provides deep, data-driven context beyond the static data views and visualizations is a structured part of a successful analytic lifecycle. There are three structural elements of data storytelling that contribute to its success:

• Data: Data represents the raw material of any narrative. Storytellers must connect the dots using insights from data to create a meaningful, compelling story for decision-makers.
• Visualization: Visualization is a way to accurately share data in the context of a narrative. Charts, graphs, and other tools “can enlighten the audience to insights that they wouldn’t see without [them],” Forbes observes, where insights might otherwise remain hidden to the untrained eye.
  
  
• Narrative: A narrative enables the audience to understand the business and emotional importance of the storyteller’s findings. A compelling narrative helps boost decision-making and instills confidence in decision-makers.

In the best cases, storytellers can craft and automate engaging, dynamic narrative reports using the very same platform they use to prepare data models and conduct advanced analytics inquiries. Processes may be automated so that storytellers can prepare data models and conduct inquiries easily as they shape their narrative. But whether the storyteller has access to a legacy or modern business intelligence (BI)platform , it’s the storyteller and his or her capabilities that matter most.

Who are your storytellers?

"The ability to take data - to be able to understand it, to process it, to extract value from it, to visualize it, to communicate it - that’s going to be a hugely important skill in the next decades."

- Hal R. Varian, Chief Economist, Google, 2009

The history of analytics has been shaped by technical experts, where companies prioritized data scientists who can identify and understand raw information and process insights themselves. But as business became more data-driven, the need for insights spread across the organization. Business success called for more nuanced approaches to analysis and required broader access to analytics capabilities.

Now, organizations more often lack the storytelling skill set - the ability to bridge the gap between analytics and business value. Successful storytellers embody this 'bridge' as a result of their ability to close the gap between analytics and business decision-makers at all levels of the organization.

Today, a person doesn’t need to be a professional data scientist to master data storytelling. 'Citizen data scientists' can master data storytelling in the context of their or their team’s decision-making roles. In fact, the best storytellers have functional roles that equip them with the right vocabulary to communicate with their peers. It’s this “last mile” skill that makes the difference between information and results.

Fortunately, leading BI platforms provide more self-service capabilities than ever, enabling even nontechnical users to access in-depth insights appropriate to their roles and skill levels. More than ever, employees across business functions can explore analytics data and hone their abilities in communicating its value to others. The question is whether or not you can trust your organization to facilitate their development.

This is the end of part 1 of this article. To continue reading, you can find part 2 here.

Author: Omri Kohl 

Source: Pyramid Analytics

Data alone is not enough, storytelling matters - part 2

This article comprises the second half of a 2 part piece. Be sure to read part 1 before reading this article.

Three common mistakes in data storytelling

Of course, there are both opportunities and risks when using narratives and emotions to guide decision-making. Using a narrative to communicate important data and its context means listeners are one-step removed from the insights analytics provide.

These risks became realities in the public discourse surrounding the 2020 global COVID-19 pandemic. Even as scientists recommended isolation and social distancing to ´flatten the curve´ - low the spread of infection - fears of an economic recession grew rampant. Public figures often overlooked inconvenient medical data in favor of narratives that might reactivate economic activity, putting lives at risk.

Fortunately, some simple insights into human behavior can help prevent large-scale mistakes. Here are three common ways storytellers make mistakes when they employ a narrative, along with a simple use case to illustrate each example:

• 'Objective' thinking: In this case, the storyteller focuses on an organizational objective instead of the real story behind the data. This might also be called cognitive bias. It’s characterized by the storyteller approaching data with an existing assumption rather than a question. The analyst therefore runs the risk of picking data that appears to validate that assumption and overlooking data that does not.
  
  Imagine a retailer who wants to beat its competitor’s customer service record. Business leaders task their customer experience professionals with proving this is the case. Resolute on meeting expectations, those analysts might omit certain data that doesn’t tip the results in favor of the desired outcome.
  
  
• 'Presentative' thinking: In this case, the storyteller focuses on the means by which he or she presents the findings - such as a data visualization method - at risk of misleading, omitting, or watering down the data. The storyteller may favor a visualization that is appealing to his or her audience at the expense of communicating real value and insights.
  
  Consider an example from manufacturing. Imagine a storyteller preparing a narrative about productivity for an audience that prefers quantitative data visualization. That storyteller might show, accurately, that production and sales have increased but omit qualitative data analysis featuring important customer feedback.
  
  
• 'Narrative' thinking: In this case, the storyteller creates a narrative for the narrative’s sake, even when it does not align well with the data. This often occurs when internal attitudes have codified a specific narrative about, say, customer satisfaction or performance.
  
  During the early days of testing for COVID-19, the ratio of critical cases to mild ones appeared high because not everyone infected had been tested. Despite the lack of data, this quickly solidified a specific media narrative about the lethality of the disease.

Business leaders must therefore focus on maximizing their 'insight-to-value conversion rate', as Forbes describes it, where data storytelling is both compelling enough to generate action and valuable enough for that action to yield positive business results. Much of this depends on business leaders providing storytellers with the right tools, but it also requires encouragement that sharing genuine and actionable insights is their top priority.

Ensuring storytelling success

“Numbers have an important story to tell. They rely on you to give them a clear and convincing voice.”

- Stephen Few, Founder & Principal, Perceptual Edge®

So how can your practical data scientists succeed in their mission: driving positive decision-making with narratives that accurately reflect the story behind the data your analytics provide? Here are some key tips to relay to your experts:

• Involve stakeholders in the narrative’s creation. Storytellers must not operate in a vacuum. Ensure stakeholders understand and value the narrative before its official delivery.
  
  
• Ensure the narrative ties directly to analytics data. Remember, listeners are a step removed from the insights your storytellers access. Ensure all their observations and visualizations have their foundations in the data.
  
  
• Provide deep context with dynamic visualizations and content. Visualizations are building blocks for your narrative. With a firm foundation in your data, each visualization should contribute honestly and purposefully to the narrative itself.
  
  
• Deliver contextualized insights. 'Know your audience' is a key tenant in professional writing, and it’s equally valuable here. Ensure your storytellers understand how listeners will interpret certain insights and findings and be ready to clarify for those who might not understand.
  
  
• Guide team members to better decisions. Ensure your storytellers understand their core objective - to contribute honestly and purposefully to better decision-making among their audience members.

As citizen data science becomes more common, storytellers and their audience of decision-makers are often already on the same team. That’s why self-service capabilities, contextual dashboards, and access to optimized insights have never been so critical to empowering all levels of the organization.

Getting started: creating a culture of successful storytelling

Insights are only valuable when shared - and they’re only as good as your team’s ability to drive decisions with them in a positive way. It’s data storytellers who bridge the gap from pure analytics insights to the cognitive and emotional capacities that regularly guide decision-making among stakeholders. As you might have gleaned from our two COVID-19 scenarios, outcomes are better when real data, accurate storytelling, and our collective capacities are aligned.

But storytellers still need access to the right tools and contextual elements to bridge that gap successfully. Increasing business users’ access to powerful analytics tools is your first step towards data storytelling success. That means providing your teams with an analytics platform that adds meaning and value to business decisions, no matter their level in your organization.

If you haven´t read part 1 of this article yet, you can find it here.

Author: Omri Kohl

Source: Pyramid Analytics

Data as a universal language

You don’t have to look very far to recognize the importance of data analytics in our world; from the weather channel using historical weather patterns to predict the summer, to a professional baseball team using on-base plus slugging percentage to determine who is more deserving of playing time, to Disney using films’ historical box office data to nail down the release date of its next Star Wars film.

Data shapes our daily interactions with everything, from the restaurants we eat at, to the media we watch and the things that we buy. Data defines how businesses engage with their customers, using website visits, store visits, mobile check-ins and more to create a customer profile that allows them to tailor their future interactions with you. Data enhances how we watch sports, such as the world cup where broadcasters share data about players’ top speed and how many miles they run during the match. Data is also captured to remind us how much time we are wasting on our mobile devices, playing online games or mindlessly scrolling through Instagram.

The demand for data and the ability to analyze it has also created an entire new course of study at universities around the world, as well as a career path that is currently among the fastest growing and most sought-after skillsets. While data scientists are fairly common and chief data officer is one of the newest executive roles focused on data-related roles and responsibilities, data analytics no longer has to be exclusive to specialty roles or the overburdened IT department. 

And really, what professional can’t benefit from actionable intelligence?

Businesses with operations across the country or around the world benefit from the ability to access and analyze a common language that drives better decision making.  An increasing number of these businesses recognize that they are creating volumes of data that have value, and even more important perhaps, the need for a centralized collection system for the information so they use the data to be more efficient and improve their chances for success.

Sales teams, regardless of their location, can use centrally aggregated customer data to track purchasing behavior, develop pricing strategies to increase loyalty, and identify what products are purchased most frequently in order to offer complementary solutions to displace competitors.

Marketing teams can use the same sales data to develop focused campaigns that are based on real experiences with their customers, while monitoring their effectiveness in order to make needed adjustments and or improve future engagement.

Inventory and purchasing can use the sales data to improve purchasing decisions, ensure inventory is at appropriate levels and better manage slow moving and dead stock to reduce the financial impact on the bottom line.

Branch managers can use the same data to focus on their own piece of the business, growing loyalty among their core customers and tracking their sales peoples’ performance.

Accounts receivables can use the data to focus their efforts on the customers that need the most attention in terms of collecting outstanding invoices. And integrating the financial data with operational data paints a more complete picture of performance for financial teams and executives responsible for reporting and keeping track of the bottom line.

Data ties all of the disciplines and departments together regardless of their locations. While some may care more about product SKUs than P&L statements or on-time-in-full deliveries, they can all benefit from a single source of truth that turns raw data into visual, easy-to-read charts, graphs and tables.

The pace, competition and globalization of business make it critical for your company to use data to your advantage, which means moving away from gut feel or legacy habits to basing key decisions on the facts found in your ERP, CRM, HR, marketing and accounting systems. With the right translator, or data analytics software, the ability to use your data based on roles and responsibilities to improve sales and marketing strategies, customer relationships, stock and inventory management, financial planning and your corporate performance, can be available to all within your organization, making data a true universal language.

Source: Phocas Software

Data governance: using factual data to form subjective judgments

Data Warehouses were born of the finance and regulatory age. When you peel away the buzz words, the principle goal of this initial phase of business intelligence was the certification of truth. Warehouses helped to close the books and analyze results. Regulations like Dodd Frank wanted to make sure that you took special care to certify the accuracy of financial results and Basel wanted certainty around capital liquidity and on and on. Companies would spend months or years developing common metrics, KPIs, and descriptions so that a warehouse would accurately represent this truth.

In our professional lives, many items still require this certainty. There can only be one reported quarterly earnings figure. There can only be one number of beds in a hospital or factories available for manufacturing. However, an increasing number of questions do not have this kind of tidy right and wrong answer. Consider the following:

• Who are our best customers?
• Is that loan risky?
• Who are our most effective employees?
• Should I be concerned about the latest interest rate hike?

Words like best, risky, and effective are subjective by their very natures. Jordon Morrow (Qlik) writes and speaks extensively about the importance of data literacy and uses a phrase that has always felt intriguing: data literacy requires the ability to argue with data. This is key when the very nature of what we are evaluating does not have neat, tidy truths.

Let’s give an example. A retail company trying to liquidate its winter inventory and has asked three people to evaluate the best target list for an e-mail campaign.

• John downloads last year’s campaign results and collects the names and e-mail addresses of the 2% that responded to the campaign last year with an order.
• Jennifer thinks about the problem differently. She looks through sales records of anyone who has bought winter merchandise in the past 5 years during the month of March who had more than a 25% discount on the merchandise. She notices that these people often come to the web site to learn about sales before purchasing. Her reasoning is that a certain type of person who likes discounts and winter clothes is the target.
• Juan takes yet another approach. He looks at social media feeds of brand influencers. He notices that there are 100 people with 1 million or more followers and that social media posts by these people about product sales traditionally cause a 1% spike in sales for the day as their followers flock to the stores. This is his target list.

So who has the right approach? This is where the ability to argue with data becomes critical. In theory, each of these people should feel confident developing a sales forecast on his or her model. They should understand the metric that they are trying to drive and they should be able to experiment with different ideas to drive a better outcome and confidently state their case.

While this feels intuitive, enterprise processes and technologies are rarely set up to support this kind of vibrant analytics effort. This kind of analytics often starts with the phrase “I wonder if…” while conventional IT and data governance frameworks are not able generally to deal with questions that a person did not know that they had 6 months before. And yet, “I wonder if” relies upon data that may have been unforeseen. In fact, it usually requires a connection of data sets that have often never been connected before to drive break-out thinking. Data science is about identifying those variables and metrics that might be better predictors of performance. This relies on the analysis of new, potentially unexpected data sets like social media followers, campaign results, web clicks, sales behavior etc. Each of these items might be important for an analysis, but in a world in which it is unclear what is and is not important, how can a governance organization anticipate and apply the same dimensions of quality to all of the hundreds of data sets that people might use? And how can they apply the same kind of rigor to data quality standards for the hundreds of thousands of data elements available as opposed to the 100-300 critical data elements.

They can’t. And that’s why we need to re-evaluate the nature of data governance for different kinds of analytics.

Author: Joe Dos Santos

Source: Qlik

Data labeling: the key to AI success

In this article, Carlos Melendez, COO, Wovenware, discusses best practices for “The Third Mile in AI Development” – the huge market subsector in data labeling companies, as they continue to come up with new ways to monetize this often-considered tedious aspect of AI development. The article addresses this trend and outlines how it is not really a commodity market, but can comprise different strategies for successful outcomes. Wovenware is a Puerto Rico-based design-driven company that delivers customized AI and other digital transformation solutions that create measurable value for government and private business customers across the U.S.

The growth of AI has spawned a huge market subsector and increasing interest among investors in data labeling. In the past year, companies specializing in data labeling have secured millions of dollars in funding and they continue to come up with new ways to monetize this often-considered tedious aspect of AI development. Yet, what can be viewed as the third mile in AI development, data labeling, is also perhaps the most crucial one to effective AI solutions.

In very general terms,  AI development can be broken down into four key phases:

• Phase 1: The design phase, where the problem is identified, the solution is designed and the success criteria is defined
• Phase 2: The data collection phase, where all the data needed to train the algorithm is gathered;
• Phase 3: The development phase, where the data is cleaned and labeled and the algorithm is developed and trained
• Phase 4: The deployment phase, where the solution is set loose to perform and then continuously updated for improvement

Data Labeling is Not Created Equal

The third mile in AI development is where the action begins. Massive amounts of data is needed to train and refine the AI model – our experience has showed us that a minimum of 10,000 labeled data points are needed –  and it must be in a structured format to test and validate it, and train the model to identify and understand recurring patterns. The labels can be in the form of boxes around objects, tagging items visually or with text labels in images or in a text-based database that accompanies the original data.

Once trained with annotated data, the algorithm can begin to recognize the same patterns in new unstructured data. To get the raw data into the shape it needs to be in, it is cleaned (errors fixed and duplicate information deleted); and labeled with its proper identification.

Much of data labeling is a manual and laborious process. It involves groups of people who must label images as “cars,” or more specifically, “white cars,” or whatever the specifics might be, so that the algorithm can go out and find them. As with many things that can take time, data labeling firms are looking for a quick fix to this process. They’re turning to automated systems to tag and identify data-sets. While automation can expedite part of the process, it needs to be kept in check to ensure that AI solutions making critical decisions are not faulty. Consider the ramifications of an algorithm trained to identify children at the cross-walk of a busy intersection not recognizing those of a certain height because the data set used to train the algorithm didn’t have data about these children.

Since data is the lifeblood to effective AI, it’s no wonder that investors are seeing huge growth opportunities for the market. Effective data labeling firms are in hot demand as companies look to find a faster path to AI transformation. To aggregate and label data not only takes months of time, but effective algorithms get better over time, so it’s a constant process. But when selecting a data labeling firm that automates the process, buyers must beware. Data labeling is not yet a commodity market, and there are many ways to approach it. Consider the following when determining how to accomplish your critical data labeling process:

• Use custom-data. There is still enormous competitive advantage to owning your own quality private data-sets, so if selecting a partner, make sure the data is quality controlled and know if synthetic data is used to enrich the data-set..
• Effective data labeling requires expertise.  Many firms will crowd source annotators, or use staff with little-to-no experience, but good data labeling requires really good eyes, as well as skill. A data labeler gets better and faster over time and learns how to avoid false positives because of bad data.
• Data privacy should remain paramount. Since effective training data requires lots of company information in many cases, those performing your data labeling should be under NDA with your firm or service provider.
• Data labelers and data scientists should be part of a single team. It’s important that a data scientist building the algorithm is overseeing the data labeling to provide quality assurance and control. They will make sure it is being trained on the best data-sets and addressing the needs specific to the goal of the AI project.
• Find a long-term partner, not a data labeling factory. Since AI is never one and done, it’s important to constantly train your algorithm to do better.  Selecting a partner who developed the original algorithm, understands it best and can use the same process to improve it is crucial to continuously improving AI.
• Partially automate when needed. While automated data labeling can be rather fast, it is nowhere as precise or effective as human-led work. Partial automation can point data labelers to where objects are, so that they only need to segment them. Leading with human intelligence, augmented by automation is always best.

As data continues to become the oil that fuels effective AI, it’s critical that getting it into shape for algorithm training is not treated as a commodity, but given the attention it deserves. Data labeling can never be a one-size-fits all task, but requires the expertise, customization, collaboration and strategic approachthat results in smarter solutions.

Source: Insidebigdata

Data management: building the bridge between IT and business

We all know businesses are trying to do more with their data, but inaccuracy and general data management issues are getting in the way. For most businesses, the status quo for managing data is not always working. However, tnew research shows that data is moving from a knee-jerk, “must be IT’s issue” conversation, to a “how can the business better leverage this rich (and expensive) data resource we have at our fingertips” conversation.

The emphasis is on “conversation”, business and IT need to communicate in the new age of Artificial Intelligence, Machine Learning and Interactive Analytics. Roles and responsibilities are blurring, and it is expected that a company’s data will quickly turn from a cost-center of IT infrastructure to a revenue-generator for the business. In order to address the issues of control and poor data quality, there needs to be an ever-increasing bridge between IT and the business. This bridge has two component parts. The first one is technology, which is both sophisticated enough to handle complex data issues but easy enough to provide a quick time-to-value. The second one is people who are able to bridge the gap between IT systems/storage/access items and business users need for value and results (enter data analysts and data engineers).

This bridge needs to be built with three key components in mind:

• Customer experience:

  For any B2C company, customer experience is the number one hot topic of the day and a primary way they are leveraging data. A new 2019 data management benchmark report found that 98% of companies use data to improve customer experience. And for good reason, between social media, digital streaming services, online retailers and others, companies are looking to show the consumer that they aren’t just a corporation, but that they are the corporation most worthy of building a relationship with. This invariably involves creating a single view of the customer (SVC), and  that view needs to be built around context and based on the needs of the specific department within the business (accounts payable, marketing, customer service, etc.).

• Trust in data:

  Possessing data and trusting data are two completely different things. Lots of companies have lots of data, but that doesn’t mean they automatically trust it enough to make business-critical decisions with it. Research finds that on average, organizations suspect 29% of current customer/prospect data is inaccurate in some way. In addition, 95% of organizations see impacts in their organization from poor quality data. A lack of trust in the data available to business users paralyzes decisions, and even worse, impacts the ability to make the right decisions based on faulty assumptions. How often have you received a report and questioned the results? More than you’d like to admit, probably. To get around this hurdle, organizations need to drive culture change around data quality strategies and methodologies. Only by completing a full assessment of data, developing a strategy to address the existing and ongoing issues, and implementing a methodology to execute on that strategy, will companies be able to turn the corner from data suspicion to data trust.

• Changing data ownership:

  The responsibilities between IT and the business are blurring. 70% of businesses say that not having direct control over data impacts their ability to meet strategic objectives. The reality is that the definitions of control are throwing people off. IT thinks of control as storage, systems, and security. The business thinks of control as access, actionable and accurate. The role of the CDO is helping to bridge this gap, bringing the nuts and bolts of IT in line with the visions and aspirations of the business.

The bottom line is that for most companies data is still a shifting sea of storage, software stacks, and stakeholders. The stakeholders are key, both from IT and the business, and in how the two can combine to provide the oxygen the business needs to survive: better customer experience, more personalization, and an ongoing trust in the data they administrate to make the best decisions to grow their companies and delight their customers.

Author: Kevin McCarthy

Source: Dataversity

Data science community to battle COVID-19 via Kaggle

A challenge on the data science community site Kaggle is asking great minds to apply machine learning to battle the COVID-19 coronavirus pandemic.

As COVID-19 continues to spread uncontrolled around the world, shops and restaurants have closed their doors, information workers have moved home, other businesses have shut down entirely, and people are social distancing and self-isolating to 'flatten the curve'. It's only been a few weeks, but it feels like forever. If you listen to the scientists, we have a way to go still before we can consider reopening and reconnecting. The worst crisis is yet to come for many areas. Yet, there are glimmers of hope, too.

Among them are the efforts of so many smart minds working on different parts of the problem to track hospital beds, map the spread, research the survivors, develop treatments, create a vaccine, and many other innovations. To help spur the development, researchers from several organizations at the request of the White House Office of Science and Technology Policy have released a dataset of machine-readable coronavirus literature for data and text mining, which includes more than 29,000 articles of which more than 13,000 have full text.

The dataset is available to researchers around the world via Google's Kaggle machine learning and data science community, the White House office announced earlier this month, and was made available from researchers and leaders from the Allen Institute for AI, Chan Zuckerberg Initiative, Georgetown University's Center for Security and Emerging Technology, Microsoft, and the National Library of Medicine at the National Institutes of Health.

Together, the White House and the organizations have issued a call to action to the nation's AI experts 'to develop new text and data mining techniques that can help the science community answer high-priority scientific questions related to COVID-19'.

Among those answering the call is data science and AI platform company DataRobot, which announced that it would provide the platform for free to those who want to use it to help with the COVID-19 virus response effort. In collaboration with its cloud partner, AWS (which has also waived its fees), the program offers free access to the DataRobot's automated machine learning and Paxata data preparation technology for those participating in the Kaggle challenge.

DataRobot has brought those 13,000 data sets into the DataRobot platform and performed some initial data preparation, Phil Gurbacki, senior VP of product and customer experience told InformationWeek. Some of the initial projects are looking at risk factors, seasonal factors, and how to identify the origin of transmission, he said. Gurbacki said the time series forecast model capabilities of the DataRobot platform could be particularly useful to data scientists looking to model impacts of the virus.

'Innovation starts with an understanding', Gurbacki said. 'We want to make sure we maximize the amount of time that researchers are spending on innovation rather than wasting time doing something that could be automated for them'.

DataRobot joins many other companies that are offering their platforms for free for a limited period as the world responds to the challenges of the novel coronavirus. GIS and mapping software company Esri is also offering its platform free of charge to those working on fighting the pandemic, particularly governments around the world. It has also built templates and a hub that spotlights notable projects.

Plus, there are several vendors that are offering free trial versions of collaboration software for organizations that are now operating with a remote workforce. Those companies include Microsoft with its Teams collaboration software, Atlassian, Cisco's Webex, Facebook for Workplace, Google Hangouts, Slack, Stack Overflow Teams, Zoho Remotely, and Zoom, among many others.

Author: Jessica Davis

Source Informationweek

Van fouten bij de Belastingdienst kunnen we veel leren

De belastingdienst verkeert opnieuw in zwaar weer. Na de negatieve berichtgeving in 2016 was in Zembla te zien hoe de belastingdienst invulling gaf aan Data Analytics. De broedkamer waarin dat gebeurde stond intern bekend als domein om te 'prutsen en pielen zonder pottenkijkers'.

Wetgeving met voeten getreden

Een overheidsdienst die privacy- en aanbestedingswetgeving met voeten treedt staat natuurlijk garant voor tumult en kijkcijfers. En terecht natuurlijk. Vanuit oorzaak en gevolg denken is het echter de vraag of die wetsovertredingen nou wel het meest interessant zijn. Want hoe kon het gebeuren dat een stel whizzkids in datatechnologie onder begeleiding van een extern bureau (Accenture) in een ‘kraamkamer’ werden gezet. En zo, apart van de gehele organisatie, een vrijbrief kregen voor…….Ja voor wat eigenlijk?

Onder leiding van de directeur van de belastingdienst Hans Blokpoel is er een groot data en analytics team gestart. Missie: alle bij de belastingdienst bekende gegevens te combineren, om zo efficiënter te kunnen werken, fraude te kunnen detecteren en meer belastingopbrengsten te genereren. En zo dus waarde voor de Belastingdienst te genereren. Dit lijkt op een data science strategie. Maar wist de belastingdienst wel echt waar ze mee bezig was? Vacatureteksten die werden gebruikt om data scientists te werven spreken van ‘prutsen en pielen zonder pottenkijkers’.

De klacht van Zembla is dat het team het niveau van ‘prutsen en pielen’ feitelijk niet ontsteeg. Fysieke beveiliging, authenticatie en autorisatie waren onvoldoende. Het was onmogelijk te zien wie bij de financiële gegevens van 11 miljoen burgers en 2 miljoen bedrijven geweest was, en of deze gedownload of gehackt waren. Er is letterlijk niet aan de wetgeving voldaan.

Problemen met data science

Wat bij de Belastingdienst misgaat gebeurt bij heel erg veel bedrijven en organisaties. Een directeur, manager of bestuurder zet data en analytics in om (letterlijk?) slimmer te zijn dan de rest. Geïsoleerd van de rest van de organisatie worden slimme jongens en meisjes zonder restricties aan de slag gezet met data. Uit alle experimenten en probeersels komen op den duur aardige resultaten. Resultaten die de belofte van de 'data driven organisatie' mogelijk moeten maken.

De case van de belastingdienst maakt helaas eens te meer duidelijk dat er voor een 'data driven organisatie' veel meer nodig is dan de vaardigheid om data te verzamelen en te analyseren. Tot waarde brengen van data vergt visie (een data science strategie), een organisatiewijze die daarop aansluit (de ene data scientist is de andere niet) maar ook kennis van de restricties. Daarmee vraagt het om een cultuur waarin privacy en veiligheid gewaarborgd worden. Voor een adequate invulling van de genoemde elementen heb je een groot deel van de ‘oude’ organisatie nodig alsmede een adequate inbedding van de nieuwe eenheid of funct

ie.

Strategie en verwachtingen

Data science schept verwachtingen. Meer belastinginkomsten met minder kosten, hogere omzet of minder fraude. Efficiency in operatie maar ook effectiviteit in klanttevredenheid. Inzicht in (toekomstige) marktontwikkelingen. Dit zijn hoge verwachtingen. Implementatie van data science vraagt echter ook om investeringen. Stevige investeringen in technologie en hoogopgeleide mensen. Schaarse mensen bovendien met kennis van IT, statistiek, onderzoeksmethodologie etc. Hoge verwachtingen die gepaard gaan met stevige investeringen leiden snel tot teleurstellingen. Teleurstellingen leiden tot druk. Druk leidt niet zelden tot het opzoeken van grenzen. En het opzoeken van grenzen leidt tot problemen. De functie van een strategie is deze dynamiek te voorkomen.

Het managen van de verhouding tussen verwachtingen en investeringen begint bij een data science strategie. Een antwoord op de vraag: Wat willen we in welke volgorde volgens welke tijdspanne met de implementatie van data science bereiken? Gaan we de huidige processen optimaliseren (business executie strategie) of transformeren (business transformatie strategie)? Of moet het data science team nieuwe wijzen van werken faciliteren (enabling strategie)? Deze vragen zou een organisatie zichzelf moeten stellen alvorens met data science te beginnen. Een helder antwoord op de strategie vraag stuurt de governance (waar moeten we op letten? Wat kan er fout gaan?) maar ook de verwachtingen. Bovendien weten we dan wie er bij de nieuwe functie moet worden betrokken en wie zeker niet.

Governance en excessen

Want naast een data science strategie vraag adequate governance om een organisatie die in staat is om domeinkennis en expertise uit het veld te kunnen combineren met data. Dat vereist het in kunnen schatten van 'wat kan' en 'wat niet'. En daarvoor heb je een groot deel van de 'oude' organisatie nodig. Lukt dat, dan is de 'data driven organisatie' een feit. Lukt het niet dan kun je wachten op brokken. In dit geval dus een mogelijke blootstelling van alle financiele data van alle 11 miljoen belastingplichtige burgers en 2 miljoen bedrijven. Een branchevreemde data scientist is als een kernfysicus die in experimenten exotische (en daarmee ook potentieel gevaarlijke) toepassingen verzint. Wanneer een organisatie niet stuurt op de doelstellingen en dus data science strategie dan neemt de kans op excessen toe.

Data science is veelmeer dan technologie

Ervaringsdeskundigen weten al lang dat data science veelmeer is dat het toepassen van moderne technologie op grote hoeveelheden data. Er zijn een aantal belangrijke voorwaarden voor succes. In de eerste plaats gaat het om een visie op hoe data en data technologie tot waarde kunnen worden gebracht. Vervolgens gaat het om de vraag hoe je deze visie organisatorisch wilt realiseren. Pas dan ontstaat een kader waarin data en technologie gericht kunnen worden ingezet. Zo kunnen excessen worden voorkomen en wordt waarde gecreëerd voor de organisatie. Precies deze stappen lijken bij de Belastingdienst te zijn overgeslagen.

Zembla

De door Zembla belichtte overtreding van wetgeving is natuurlijk een stuk spannender. Vanuit het credo ‘voorkomen is beter dan genezen’ blijft het jammer dat het goed toepassen van data science in organisaties in de uitzending is onderbelicht.

Bron: Business Data Science Leergang Radboud Management Academy http://www.ru.nl/rma/leergangen/bds/

Auteurs: Alex Aalberts / Egbert Philips

Data science plays key role in COVID-19 research through supercomputers

Supercomputers, AI and high-end analytic tools are each playing a key role in the race to find answers, treatments and a cure for the widespread COVID-19.

In the race to flatten the curve of COVID-19, high-profile tech companies are banking on supercomputers. IBM has teamed up with other firms, universities and federal agencies to launch the COVID-19 High Performance Computing Consortium.

This consortium has brought together massive computing power in order to assist researchers working on COVID-19 treatments and potential cures. In total, the 16 systems in the consortium will offer researchers over 330 petaflops, 775,000 CPU cores and 34,000 GPUs and counting.

COVID-19 High performance computing consortium

The consortium aims to give supercomputer access to scientists, medical researchers and government agencies working on the coronavirus crisis. IBM said its powerful Summit supercomputer has already helped researchers at the Oak Ridge National Laboratory and the University of Tennessee screen 8,000 compounds to find those most likely to bind to the main "spike" protein of the coronavirus, rendering it unable to infect host cells.

"They were able to recommend the 77 promising small-molecule drug compounds that could now be experimentally tested," Dario Gil, director of IBM Research, said in a post. "This is the power of accelerating discovery through computation."

In conjunction with IBM, the White House Office of Science and Technology Policy, the U.S. Department of Energy, the National Science Foundation, NASA, nearly a dozen universities, and several other tech companies and laboratories are all involved.

The work of the consortium offers an unprecedented back end of supercomputer performance that researchers can leverage while using AI to parse through massive databases to get at the precise information they're after, Tim Bajarin, analyst and president of Creative Strategies, said.

Supercomputing powered by sharing big databases

Bajarin said that the world of research is fundamentally done in pockets which creates a lot of insulated, personalized and proprietary big databases.

"It will take incredible cooperation for Big Pharma to share their research data with other companies in an effort to create a cure or a vaccine," Bajarin added.

Gil said IBM is working with consortium partners to evaluate proposals from researchers around the world and will provide access to supercomputing capacity for the projects that can have the most immediate impact.

Many enterprises are coming together to share big data and individual databases with researchers.  

Signals Analytics released a COVID-19 Playbook that offers access to critical market intelligence and trends surrounding potential treatments for COVID-19. The COVID-19 Playbook is available at no cost to researchers looking to monitor vaccines that are in development for the disease and other strains of coronavirus, monitor drugs that are being tested for COVID-19 and as a tool to assess which drugs are being repurposed to help people infected with the virus.

"We've added a very specific COVID-19 offering so researchers don't have to build their own taxonomy or data sources and can use it off the shelf," said Frances Zelazny, chief marketing officer at Signals Analytics.

Eschewing raw computing power for targeted, critical insights

With the rapid spread of the virus and the death count rising, treatment options can't come soon enough. Raw compute power is important, but perhaps equally as crucial is being able to know what to ask and quickly analyze results.

"AI can be a valuable tool for analyzing the behavior and spread of the coronavirus, as well as current research projects and papers that might provide insights into how best to battle COVID-19," Charles King, president of the Pund-IT analysis firm, said.

The COVID-19 consortium includes research requiring complex calculations in& epidemiology and bioinformatics. While the high computing power allows for rapid model testing and large data processing, the predictive analytics have to be proactively applied to health IT.

Dealing with COVID-19 is about predicting for the immediate, imminent future - from beds necessary in ICUs to social distancing timelines. In the long term, Bajarin would like to see analytic and predictive AI used as soon as possible to head off future pandemics.

"We've known about this for quite a while - COVID-19 is a mutation of SARS. Proper trend analysis of medical results going forward could help head off the next great pandemic," Bajarin said.

Author: David Needle

Source: TechTarget

Data wrangling in SQL: 5 recommended methods

Data wrangling is an essential job function for data engineering, data science, or machine learning roles. As knowledgable coders, many of these professionals can rely on their programming skills and help from libraries like Pandas to wrangle data. However, it can often be optimal to manipulate data directly at the source with SQL scrips. Here are some data wrangling techniques every data expert should know.

Note for the purpose of this blog post I will not go into the details of the many versions of SQL. Much of the syntax mentioned is conformant with the SQL-92 standard, but where there are platform-specific clauses and functions, I will try and point them out (PostgreSQL, MySQL, T-SQL, PSQL, etc) but not extensively. Remember, however, it’s common to implement a missing function directly from your favorite flavor of SQL. For example, the correlation coefficient function, CORR, is present in PostgreSQL and PLSQL but not MySQL and can be implemented directly or imported from an external library. To keep this blog brief, I don’t specify the implementation of these techniques but they are widely available and will also be covered in my upcoming virtual training session where I’ll go into further detail on data wrangling techniques and how to wrangle data.

1. Data Profiling with SQL

Data profiling is often the first step in any data science or machine learning project and an important preliminary step to explore the data prior to data preparation or transformation. The goal is straightforward: determine if the data is accurate, reliable, and representative to build performing algorithms since the resulting models are only as good as the data employed therein.

Core SQL Skills for Data Profiling

• Descriptive statistics: MIN, MAX, MEAN, AVERAGE, MEDIAN,, STDEV, , STDDEV(PostgreSQL), STDEVP (Transact-SQL)
• Correlation functions:: CORR (PostgreSQL), 
• Aggregation functions: COUNT, SUM, GROUP BY

Data profiling starts with some basic descriptive statistic functions such as MIN, MAX, STDEV, etc to help you understand the distribution and range of your data. Even basic aggregation functions like COUNT and GROUP BY are useful. From there, standard deviation functions including STDEV/STDEVP can tell you a lot about your data (high, low) and its distribution. For a quick look at correlations across features and the target feature, the CORR correlation coefficient function can give some quick insight. 

2. Detecting Outliers with SQL

An essential part of data profiling is identifying outliers, and SQL can be a quick and easy way to find extreme values. You can get quite far with the below simple techniques and employing basic sorting techniques.

Core SQL Skills for Detecting Outliers 

• Sorting & Filtering: ORDER, CASE, WHERE HAVING
• Descriptive statistics: AVG, AVERAGE, MEAN, MEDIAN
• Winsorization: PERCENTILE_CONT, PERCENT_RANK(T-SQL),  Subqueries, VAR (T-SQL) 
• Fixing outliers: UPDATE, NULL, AVG

Simple sorting and ordering using ORDER BY will produce quick results and further classifying using CASE statements and using groups and summation. More sophisticated techniques can be used by implementing various forms of Winsorization and variance functions VAR(T-SQL). To identify outliers you can use percentile calculations, and VAR(T-SQL) can be used to measure the statical variants.

There are various techniques for handling outliers from ignoring to removing (trimming). Replacement techniques such as Winsorization where we essentially change extreme values in the dataset to less extreme values with observations closest to them. This typically requires us to calculate percentiles. We can use UPDATE with PERCENTILE_CONT (present in PostgreSQL, T-SQL, and PLSQ) to, for example,  sets all observations greater than the 95th percentile equal to the value at the 95th percentile and all observations less than the 5th percentile equal to the value at the 5th percentile.

3. Time Series, ReIndexing, and Pivoting with SQL

Time-series databases such as Prometheus, TDengine, and InfluxDB are quite popular among data wrangling techniques but more often than not you may be dealing with a standard relational database, and pivoting, reindexing, or transforming data into time series can be a challenge.

Core SQL Skills for TimeSeries, ReIndexing, and Pivoting with SQL

• Window Functions:  LAG, OVER, (PostgreSQL, T-SQL, MySQL)   OVER
• Time-series generation: Pivoting 
• PIVOT (T-SQL) CROSSTAB, CASE
• Data Trends: COALSE

For time series and data realignment, window functions perform calculations on a set of rows that are related together and thus perfect for this type of data transformation. For example, if we have a set of time-stamped data and we wanted to get the difference at each step, you can use the LAG function. The LAG() function provides access to a row that comes before the current row. This works because window operations like LAG() do not collapse groups of query rows into a single output row. Instead, they produce a result for each row. Using the OVER clause specifies you can  partition query rows into groups for processing by the window function: 

Reindexing (not to be confused with database indexing) or aligning datasets is another key technique. For example, you have sales from different time periods you’d like to compare. Using windowing functions and subqueries you can reset the data from the same data point or zero or align on the same period units in time.

Pivoting is a widely used spreadsheet technique among data wrangling techniques. Often when looking at periodic or timestamp data you get a single column of data you’d like to pivot to multispoke columns. For example, pivoting a single column of monthly sales listed by rows into multiple columns of sales by month. PIVOT and CROSSTAB can be used but for other SQL favors that don’t support those functions, you can implement them with CASE statements.

4. Fake Data Generation with SQL

There are a number of reasons to generate fake data, including:

• Anonymizing production and sensitive data before porting it to other data stores
• Generating more data that has similar characteristics to existing data for load testing, benchmarking, robustness testing, or scaling. 
• Varying the distribution of your data to understand how your data scienceor ML model performs for bias, loss function, or other metrics

Core SQL skills:

• RANDOM, RAND
• CREATE, UPDATE, SELECT, INSERT, and subqueries
• JOIN and CROSS JOIN, MERGE CAST
• String manipulation: functions, CONCAT SUBSTR, TRIM, etc. 

SQL can be useful for generating fake data for load testing, benchmarking, or model training and testing. For numerical data, RANDOM, RAND(), and its variants are classic random generators. Generation data for new tables (CREATE) or updating existing ones (UPDATE). Data selection and subqueries are generally a prerequisite. CROSS JOIN is a favorite as it lets you combine data (aka cartesian join) to make new data sets and MEGE can be used similarly. CAST is useful for converting variables into different data types. When generating text string manipulation  CONCAT SUBSTR, TRIM and a host of other functions can be used to create new text variables.

5. Loading Flat Files with SQL 

This may look like a bonus round data wrangling technique but I’ve worked with many data experts that struggle to load data files directly into a database (the L in ETL). Why do this?  For many research projects data is not readily available in data lakes, APIs, and other data stores. Generally, it has to be collected in the wild and more often than not in a flat-file format or multiple flat files.

Core Skills required

• SQL Clients; Understand delimiters, text delimiting, and special characters
• Table creation CREATE, TEMPORARY TABLE  
• Importing data, IMPORT, INSERT, SELECT INTO
• Normalize data with SQL clauses including, SELECT, UPDATE, WHERE, JOIN, and subqueries
• Transaction control: BEGIN,  COMMIT, and ROLLBACK

ETL (Extraction, Transformation, Loading) is the most common way to simply import CSV files. You can use SQL for the all-important verification step. Simple verification includes checking for nulls, understanding the range and distribution of values imported, finding extreme values, and event simple count checks. 

However, loading flat file data into a database is not as trivial as it seems thanks to the need for specific delimiters. Most SQL clients allow you to specify custom delimiters. For text-heavy files, you can’t use simple text quote delimiters. Double quotes and other techniques must be employed. Be careful to check for whitespace characters also including newline characters.

Importing a single flat file like a CSV file is somewhat straightforward once the above steps are done. However, by nature flat files are not normalized thus you can use SQL to create a temporary table to load the data and then use the SQL filter clause and Update clause to port that data to the final destination tables. A common technique is to import that data to temporary tables in “raw” format and then normalize the data to other tables. As always when inserting or updating data employ transaction controls.

Author: Seamus McGovern

Source: Open Data Science

Dataiku door Snowflake benoemd tot Data Science Partner of the Year

De impact van 5G op de ontwikkelingen in moderne technologie

Van opvouwbare 5G-telefoons tot operaties die worden uitgevoerd op een patiënt op kilometers afstand, en van vroege supersnelle netwerkuitrol in de VS tot tactiele internet en Internet of Things, en natuurlijk robots... het lijkt wel alsof 5G altijd en overal onderwerp van gesprek is.

Voor telecom operators of communicatie-providers (CSP's) is dit een tijd vol kansen, van het upgraden van capaciteit tot het leveren van nieuwe services, content en interactie op manieren die voorheen simpelweg niet mogelijk waren. Door 5G te gebruiken om Internet of Things (IoT) en edge computing mogelijk te maken, hebben ze nu een geweldige kans die een paar jaar geleden nog volledig ondenkbaar was.

Maar dat betekent dat 5G zich op een enorm buigpunt bevindt. Van het kunnen verslaan van concurrenten met nieuwe diensten die echt waarde toevoegen, tot de prestaties en operationele efficiëntie die hun netwerken aanzienlijk verbeteren. De beslissingen die vandaag worden genomen, zullen verregaande financiële en operationele implicaties hebben voor CSP's.

Bereik bedrijfsdoelen met 5G

Maar als je 5G slechts ziet als een volgende mogelijkheid voor telecom, dan mis je de boot. Iedere organisatie in iedere sector die netwerken gebruikt (kortom iedereen) moet plannen maken voor de 5G implementatie en nadenken over hoe ze deze kunnen gebruiken om hun bedrijfsdoelen te bereiken.

Het maakt niet uit of ze actief zijn in de detailhandel, logistiek, in een stad, in een landelijke omgeving, in de publieke of private sector. 5G belooft enorme kansen om nieuwe diensten en toepassingen te leveren, de automatisering en de mogelijkheden die dit met zich meebrengt te vergroten en om bedrijven te helpen om met klanten om te gaan op manieren die nog nooit eerder waren bedacht.

Dat wil niet zeggen dat het altijd even gemakkelijk is, zoals Åsa Tamsons, hoofd new businesses bij Ericsson, al zei in een interview met CN. Want velen beschouwen 5G nog steeds als gewoon 'een ander netwerk'. Het is een houding die het aanvankelijk voor sommige organisaties moeilijker zou kunnen maken om de vooraf benodigde investeringen rond te krijgen. Ondernemingen zullen moeten werken om zowel interne als externe belanghebbenden ervan te overtuigen dat de kosten gerechtvaardigd zijn.

Eén netwerk, een wereld van kansen

Terug naar de initiële vraag: is 5G niet gewoon een andere G? Simpel gezegd, 5G levert veel hogere snelheden en biedt veel kortere latency en een aanzienlijk hogere dichtheid dan 4G. Maar wat betekent dit eigenlijk?

Snelheid is relatief eenvoudig. Het 5G-voorbeeld dat hierin vaak wordt gebruikt, is dat het straks mogelijk is om in tien seconden een HD-film te downloaden in vergelijking met de (op zijn best) ongeveer twintig minuten die er momenteel voor nodig zijn (afhankelijk van de lokale breedbanddiensten).

De latency, de tijd die nodig is om gegevens tussen twee punten te laten reizen, is bij 5G minder dan een milliseconde, wat bijvoorbeeld belangrijk kan zijn voor chirurgie, maar gecombineerd met snelheid is het ook een factor voor veel gamers die willen betalen voor dit type snelle, low latency-service.

Op dit moment bevinden we ons al in een wereld waar meer dan 23 miljard apparaten zijn aangesloten op netwerken en die dichtheid blijft, dankzij grotere mobiliteit en IoT-use-cases, groeien. We zijn allemaal al eens in de situatie geweest waarin de snelheden drastisch afnemen als iedereen inlogt en naarmate we steeds meer verbonden raken, hebben we netwerken nodig die geschikt zijn voor aanzienlijk meer apparaten dan ooit tevoren.

Telco-cloud

Echter, om dit alles te kunnen leveren, is een aanzienlijke investering in de netwerkinfrastructuur nodig. Voor CSP's is dit een grote onderneming. Daarom is het waarschijnlijk zo dat in plaats van een puur 5G-netwerk, de meerderheid van de mensen een gemengde aanpak zullen zien, waarbij 4G beschikbaar is om basisdiensten te leveren en 5G wordt geïntroduceerd voor specifieke taken. Het is daarom van cruciaal belang om de zogenaamde telco-cloud te hebben. Dit is een software defined technologie die zowel de huidige 4G ondersteunt als het grondwerk voor 5G is, iets wat erg wordt gewaardeerd door operators zoals bijvoorbeeld Vodafone.

'Het vermogen om flexibel en agile te zijn terwijl we onze netwerkactiviteiten en -beheer blijven automatiseren, kon alleen worden bereikt door een software defined infrastructuur', zegt Johan Wibergh, chief technology officer van Vodafone. 'We zijn blij met de versnelde time-to-market en de bijbehorende economische voordelen van onze transitie naar NFV en, in toenemende mate, een telco-cloud infrastructuur'.

Met 5G hebben bedrijven toegang tot de niveaus en snelheden van connectiviteit die ze nodig hebben om te profiteren van de game changing technologieën zoals IoT, edge computing en AI (artificial intelligence) die de volgende fase van de digitale revolutie gaan vormgeven. Gecombineerd met deze software defined-infrastructuur, en meer in overeenstemming met de specificaties en ambities, heeft 5G de kracht om bedrijfsmodellen van gevestigde organisaties ongekend te transformeren.

Kapitaliseren om te gedijen

We beseffen ons nog niet eens wat de mogelijkheden van 5G nu al zijn. Er moet nog zoveel gebeuren voordat we volledige acceptatie zien, maar bedrijven moeten nu gaan nadenken over hoe ze de kracht van deze nieuwe netwerken kunnen benutten voor hun eigen concurrentievermogen. Er over denken als 'gewoon een andere G' dreigt er voor te zorgen dat men niet voorbereid is en de enorme kansen mist die beschikbaar zijn.

5G is het netwerk en de basis die de beloftes van veel andere nieuwe technologieën waar gaat maken. Elke organisatie die er niet in slaagt om hiervan te profiteren, zal heel hard moeten werken om te overleven in de digitale wereld.

Auteur: Jean Pierre Brulard

Bron: CIO

De uitdaging van het structuur aanbrengen in ongestructureerde data

De wereld verzamelt steeds meer data, en met een onrustbarend groeiende snelheid. Vanaf het begin van de beschaving tot ongeveer 2003 produceerde de mensheid zo’n 5 exabyte aan data. Nu produceren we deze hoeveelheid elke twee dagen. 90 procent van alle data is in de afgelopen 2 jaar gegenereerd.

Op zich is er niets mis met data, maar het probleem is dat een groot deel hiervan ongestructureerd is. Deze ‘dark data’ omvat inmiddels al zo’n vier vijfde van de totale databerg. En daarmee beginnen de echte problemen.

Privacy

Ongestructureerde data is onbruikbaar. Je weet niet wat erin zit, wat de structuur is en hoeveel informatie daarvan misschien belangrijk is. Hoe kun je voldoen aan de eisen van de nieuwe privacywetgeving, als je niet eens weet welke informatie er in je data zit? Het kan gevoelige informatie zijn, zodat je de wet overtreedt zonder dat je daarvan op de hoogte bent. Totdat zich een lek voordoet en alle gegevens op straat liggen. En hoe kun je voldoen aan de wet openbaarheid bestuur en straks aan de wet open overheid, als je niet weet waar je de informatie moet vinden? De AVG verplicht je om persoonsgegevens te vernietigen als de persoon daarom vraagt. Maar als je niet weet waar je die moet vinden, sta je met de mond vol tanden.

Databerg

Stel je data voor als een ijsberg. Het grootste deel ligt onder water: je ziet het niet. Wat boven het water uitsteekt is de kritische informatie die je dagelijks gebruikt en die nodig is om jouw organisatie te laten werken. Direct onder het oppervlak ligt een groot deel dat ooit kritisch was. Het is gebruikt en daarna opgeslagen om vervolgens nooit meer aangeraakt te worden: redundant, overbodig en triviaal, kortom ROT.

Het grootste deel van de berg bevindt zich daar weer onder, het is de ‘dark data’, verzameld door mensen, machines en allerlei werkprocessen. Je hebt geen idee wat er zich in dat donkere deel schuilhoudt. Het zijn gegevens die zijn verzameld door sensoren, video’s van beveiligingscamera’s, en vele, vele documenten van lang, lang geleden.

Nieuwe inzichten

Je kunt het natuurlijk negeren, je hebt het immers niet nodig voor je dagelijkse workflow. Maar voor hetzelfde geld bevindt zich in die dark data waardevolle informatie die gebruikt kan worden om de processen in de organisatie beter te laten verlopen. Of nieuwe toepassingen mogelijk te maken. Door data uit de berg te leggen op andere data bijvoorbeeld, kun je plotseling nieuwe inzichten verkrijgen waarmee beleid kan worden gemaakt: informatiegestuurd beleid.

Digitale dompteur

Als alle plannen en elke beleidsmaatregel kunnen worden onderbouwd met keiharde gegevens uit de databerg, dan hebben we de heilige graal gevonden. De kwaliteit van de dienstverlening van de overheid gaat met sprongen omhoog, en er komen nieuwe impulsen voor veiligheid, handhaving, onderhoud en schuldhulpverlening, om maar eens een paar beleidsterreinen te noemen.

Dat zal waarschijnlijk een onbereikbaar ideaal blijven, maar we kunnen wel flinke stappen in de goede richting maken. Digitaal werken betekent voortdurend aanpassen, herordenen, migreren. Om digitale informatie te temmen is een digitale dompteur nodig: een beheeromgeving die structuur aanbrengt en die inspeelt op de voortdurende veranderingen die digitalisering met zich meebrengt.

Bron: Managementbase

Deciphering the Influence of Artificial Intelligence in Today's Business Environment

You probably interact with artificial intelligence (AI) on a daily basis and don’t even realize it.

Many people still associate AI with science-fiction dystopias, but that characterization is waning as AI develops and becomes more commonplace in our daily lives. Today, artificial intelligence is a household name – and sometimes even a household presence (hi, Alexa!).

While acceptance of AI in mainstream society is a new phenomenon, it is not a new concept. The modern field of AI came into existence in 1956, but it took decades of work to make significant progress toward developing an AI system and making it a technological reality.

In business, artificial intelligence has a wide range of uses. In fact, most of us interact with AI in some form or another on a daily basis. From the mundane to the breathtaking, artificial intelligence is already disrupting virtually every business process in every industry. As AI technologies proliferate, they are becoming imperative to maintain a competitive edge.

What is AI?

Before examining how AI technologies are impacting the business world, it’s important to define the term. “Artificial intelligence” is a broad term that refers to any type of computer software that engages in humanlike activities – including learning, planning and problem-solving. Calling specific applications “artificial intelligence” is like calling a car a “vehicle” – it’s technically correct, but it doesn’t cover any of the specifics. To understand what type of AI is predominant in business, we have to dig deeper.

Machine learning

Machine learning is one of the most common types of AI in development for business purposes today. Machine learning is primarily used to process large amounts of data quickly. These types of AIs are algorithms that appear to “learn” over time.

If you feed a machine-learning algorithm more data its modeling should improve. Machine learning is useful for putting vast troves of data – increasingly captured by connected devices and the Internet of Things – into a digestible context for humans.

For example, if you manage a manufacturing plant, your machinery is likely hooked up to the network. Connected devices feed a constant stream of data about functionality, production and more to a central location. Unfortunately, it’s too much data for a human to ever sift through; and even if they could, they would likely miss most of the patterns. 

Machine learning can rapidly analyze the data as it comes in, identifying patterns and anomalies. If a machine in the manufacturing plant is working at a reduced capacity, a machine-learning algorithm can catch it and notify decision-makers that it’s time to dispatch a preventive maintenance team.

But machine learning is also a relatively broad category. The development of artificial neural networks – an interconnected web of artificial intelligence “nodes” – has given rise to what is known as deep learning.

Deep learning

Deep learning is an even more specific version of machine learning that relies on neural networks to engage in what is known as nonlinear reasoning. Deep learning is critical to performing more advanced functions – such as fraud detection. It can do this by analyzing a wide range of factors at once.

For instance, for self-driving cars to work, several factors must be identified, analyzed and responded to simultaneously. Deep learning algorithms are used to help self-driving cars contextualize information picked up by their sensors, like the distance of other objects, the speed at which they are moving and a prediction of where they will be in 5-10 seconds. All this information is calculated at once to help a self-driving car make decisions like when to change lanes.

Deep learning has a great deal of promise in business and is likely to be used more often. Older machine-learning algorithms tend to plateau in their capability once a certain amount of data has been captured, but deep learning models continue to improve their performance as more data is received. This makes deep learning models far more scalable and detailed; you could even say deep learning models are more independent.

AI and business today

Rather than serving as a replacement for human intelligence and ingenuity, artificial intelligence is generally seen as a supporting tool. Although AI currently has a difficult time completing commonsense tasks in the real world, it is adept at processing and analyzing troves of data much faster than a human brain could. Artificial intelligence software can then return with synthesized courses of action and present them to the human user. In this way, we can use AI to help game out pfossible consequences of each action and streamline the decision-making process.

“Artificial intelligence is kind of the second coming of software,” said Amir Husain, founder and CEO of machine-learning company SparkCognition. “It’s a form of software that makes decisions on its own, that’s able to act even in situations not foreseen by the programmers. Artificial intelligence has a wider latitude of decision-making ability as opposed to traditional software.”

Those traits make AI highly valuable throughout many industries – whether it’s simply helping visitors and staff make their way around a corporate campus efficiently, or performing a task as complex as monitoring a wind turbine to predict when it will need repairs.

Common uses of AI

Some of the most standard uses of AI are machine learning, cybersecurity, customer relationship management, internet searches and personal assistants.

Machine learning

Machine learning is used often in systems that capture vast amounts of data. For example, smart energy management systems collect data from sensors affixed to various assets. The troves of data are then contextualized by machine-learning algorithms and delivered to your company’s decision-makers to better understand energy usage and maintenance demands.

Cybersecurity

Artificial intelligence is even an indispensable ally when it comes to looking for holes in computer network defenses, Husain said. Believe it or not, AI systems can recognize a cyberattack, as well as other cyberthreats, by monitoring patterns from data input. Once it detects a threat, it can backtrack through your data to find the source and help  to prevent a future threat. That extra set of eyes – one that is as diligent and continuous as AI – will serve as a great benefit in preserving your infrastructure.

“You really can’t have enough cybersecurity experts to look at these problems, because of scale and increasing complexity,” Husain added. “Artificial intelligence is playing an increasing role here as well.”

Customer relationship management

Artificial intelligence is also changing customer relationship management (CRM) systems. Software programs like Salesforce and Zoho require heavy human intervention to remain current and accurate. But when you apply AI to these platforms, a normal CRM system transforms into a self-updating, auto-correcting system that stays on top of your relationship management for you.

A great example of how AI can help with customer relationships is demonstrated in the financial sector. Dr. Hossein Rahnama, founder and CEO of AI concierge company Flybits and visiting professor at the Massachusetts Institute of Technology, worked with TD Bank to integrate AI with regular banking operations.

“Using this technology, if you have a mortgage with the bank and it’s up for renewal in 90 days or less … if you’re walking by a branch, you get a personalized message inviting you to go to the branch and renew purchase,” Rahnama said. “If you’re looking at a property for sale and you spend more than 10 minutes there, it will send you a possible mortgage offer. Internet and data research

Artificial intelligence uses a vast amount of data to identify patterns in people’s search behaviors and provide them with more relevant information regarding their circumstances. As people use their devices more, and as the AI technology becomes even more advanced, users will have a more customizable experience. This means the world for your small businesses, because you will have an easier time targeting a very specific audience.

“We’re no longer expecting the user to constantly be on a search box Googling what they need,” Rahnama added. “The paradigm is shifting as to how the right information finds the right user at the right time.”

Digital personal assistants

Artificial intelligence isn’t just available to create a more customized experience for your customers. It can also transform the way your company operates from the inside. AI bots can be used as personal assistants to help manage your emails, maintain your calendar and even provide recommendations for streamlining processes.

You can also program these AI assistants to answer questions for customers who call or chat online. These are all small tasks that make a huge difference by providing you extra time to focus on implementing strategies to grow the business.

The future of AI

How might artificial intelligence be used in the future? It’s hard to say how the technology will develop, but most experts see those “commonsense” tasks becoming even easier for computers to process. That means robots will become extremely useful in everyday life.

“AI is starting to make what was once considered impossible possible, like driverless cars,” said Russell Glenister, CEO and founder of Curation Zone. “Driverless cars are only a reality because of access to training data and fast GPUs, which are both key enablers. To train driverless cars, an enormous amount of accurate data is required, and speed is key to undertake the training. Five years ago, the processors were too slow, but the introduction of GPUs made it all possible.”

Glenister added that graphic processing units (GPUs) are only going to get faster, improving the applications of artificial intelligence software across the board.

“Fast processes and lots of clean data are key to the success of AI,” he said.

Dr. Nathan Wilson, co-founder and CTO of Nara Logics, said he sees AI on the cusp of revolutionizing familiar activities like dining. Wilson predicted that AI could be used by a restaurant to decide which music to play based on the interests of the guests in attendance. Artificial intelligence could even alter the appearance of the wallpaper based on what the technology anticipates the aesthetic preferences of the crowd might be.

If that isn’t far out enough for you, Rahnama predicted that AI will take digital technology out of the two-dimensional, screen-imprisoned form to which people have grown accustomed. Instead, he foresees that the primary user interface will become the physical environment surrounding an individual.

“We’ve always relied on a two-dimensional display to play a game or interact with a webpage or read an e-book,” Rahnama said. “What’s going to happen now with artificial intelligence and a combination of [the Internet of Things] is that the display won’t be the main interface – the environment will be. You’ll see people designing experiences around them, whether it’s in connected buildings or connected boardrooms. These will be 3D experiences you can actually feel.” 

What does AI mean for the worker?

With all these new AI uses comes the daunting question of whether machines will force humans out of work. The jury is still out: Some experts vehemently deny that AI will automate so many jobs that millions of people find themselves unemployed, while other experts see it as a pressing problem.

“The structure of the workforce is changing, but I don’t think artificial intelligence is essentially replacing jobs,” Rahnama said. “It allows us to really create a knowledge-based economy and leverage that to create better automation for a better form of life. It might be a little bit theoretical, but I think if you have to worry about artificial intelligence and robots replacing our jobs, it’s probably algorithms replacing white-collar jobs such as business analysts, hedge fund managers and lawyers.”

While there is still some debate on how, exactly, the rise of artificial intelligence will change the workforce, experts agree there are some trends we can expect to see.

Will AI create jobs?

Some experts believe that, as AI is integrated into the workforce, it will actually create more jobs – at least in the short term.

Wilson said the shift toward AI-based systems will likely cause the economy to add jobs that facilitate the transition.

“Artificial intelligence will create more wealth than it destroys,” he said, “but it will not be equitably distributed, especially at first. The changes will be subliminally felt and not overt. A tax accountant won’t one day receive a pink slip and meet the robot that is now going to sit at her desk. Rather, the next time the tax accountant applies for a job, it will be a bit harder to find one.”

Wilson said he anticipates that AI in the workplace will fragment long-standing workflows, creating many human jobs to integrate those workflows.

What about after the transition?

First and foremost, this is a transition that will take years – if not decades – across different sectors of the workforce. So, these projections are harder to identify, but some other experts like Husain are worried that once AI becomes ubiquitous, those additional jobs (and the ones that had already existed) may start to dwindle.

Because of this, Husain said he wonders where those workers will go in the long term. “In the past, there were opportunities to move from farming to manufacturing to services. Now, that’s not the case. Why? Industry has been completely robotized, and we see that automation makes more sense economically.”

Husain pointed to self-driving trucks and AI concierges like Siri and Cortana as examples, stating that as these technologies improve, widespread use could eliminate as many as 8 million jobs in the U.S. alone.

“When all these jobs start going away, we need to ask, ‘What is it that makes us productive? What does productivity mean?'” he added. “Now we’re confronting the changing reality and questioning society’s underlying assumptions. We must really think about this and decide what makes us productive and what is the value of people in society. We need to have this debate and have it quickly, because the technology won’t wait for us.”

A shift to more specialized skills

As AI becomes a more integrated part of the workforce, it’s unlikely that all human jobs will disappear. Instead, many experts have begun to predict that the workforce will become more specialized. These roles will require a higher amount of that which automation can’t (yet) provide – like creativity, problem-solving and qualitative skills.

Essentially, there is likely to always be a need for people in the workforce, but their roles may shift as technology becomes more advanced. The demand for specific skills will shift, and many of these jobs will require a more advanced, technical skill set.

AI is the future

Whether rosy or rocky, the future is coming quickly, and artificial intelligence will certainly be a part of it. As this technology develops, the world will see new startups, numerous business applications and consumer uses, the displacement of certain jobs and the creation of entirely new ones. Along with the Internet of Things, artificial intelligence has the potential to dramatically remake the economy, but its exact impact remains to be seen.

Date: April 18, 2024

Author: Adam Uzialko

Source: Business News Daily

Deepfake Neural Networks: What are GANs?

Generative adversarial networks (GANs) are one of the newer machine learning algorithms that data scientists are tapping into. When I first heard it, I wondered how can networks be adversarial? I envisioned networks with swords drawn going at it. Close… but I can assure you that no networks were harmed in the making of this article.

Let’s break GAN down further to understand how this algorithm works and dispel the mystery behind it.

• Generative model: A statistical model that can generate new data. This includes the distribution of the data.
• Adversarial training process: There are two networks involved in training. One network generates the data (the generator) while the other network tries to discriminate (the discriminator) if that data is real or fake. If it is deemed to be fake, the generator is notified and tries to improve on the next batch of generated data. Therefore, the two networks are training against each other, hence the adversarial part.
• Deep learning Networks: Deep learning methods use neural network architectures to process data, which is why they are often referred to as deep neural networks.

Why on earth would you want to use a GAN?

Now that you know what a GAN is, what do you do with it? You may have heard of deepfakes and enjoyed seeing videos of political leaders uttering some unbelievable statements. (Somedays, I wonder how we would know the difference!) Other than playing tricks on the world, GANs do have a valuable purpose.

Deep learning models are data-hungry. What if you could just snap your fingers and grow your training data set? Well, GANs can help you create synthetic data for those deep learning models. Synthetic data, or artificial data, serves as proxy data because it maintains the statistical characteristics of the real-world data that it is based off. Synthetic data should generate observations based on existing variable distributions and preserve correlations amongst the variables in the data set.

Deepfakes typically use image data and the type of GAN to create synthetic image data is called a styleGAN. However, other types of data such as tabular data (think rows and columns of integers, text, etc.) can also be created. This is a tabular GAN.

I see lots of potential with GANs and synthetic data. Synthetic data allows you to create deep learning models when you may not have previously been able to do so. There simply may not be the volume of data available that is required, especially when you are working with new products or processes. Data may also be expensive and time-consuming to acquire from third-party resources or through data collection methods such as surveys and studies. Synthetic data may also help fulfill the gaps in underrepresented groups such as customer segments, regions, or even the different driving conditions required by computer vision models for self-driving cars. Lastly, because this data is generated, it does not impact human privacy (think GDPR and personal data sharing regulations) and is less risky should the data be breached.

To remind us, that while synthetic data has the potential to help us progress with deep learning, the patterns in the synthetic data must be representative of the real data and should be verified as an initial step in the modeling process. 

Author: Susan Kahler

Source: Open Data Science

Determining the feature set complexity

Thoughtful predictor selection is essential for model fairness

One common AI-related fear I’ve often heard is that machine learning models will leverage oddball facts buried in vast databases of personal information to make decisions impacting lives. For example, the fact that you used Arial font in your resume, plus your cat ownership and fondness for pierogi, will prevent you from getting a job. Associated with such concerns is fear of discrimination based on sex or race due to this kind of inference. Are such fears silly or realistic? Machine learning models are based on correlation, and any feature associated with an outcome can be used as a decision basis; there is reason for concern. However, the risks of such a scenario occurring depend on the information available to the model and on the specific algorithm used. Here, I will use sample data to illustrate differences in incorporation of incidental information in random forest vs. XGBoost models, and discuss the importance of considering missing information, appropriateness and causality in assessing model fairness.

Feature choice — examining what might be missing as well as what’s included– is very important for model fairness. Often feature inclusion is thought of only in terms of keeping or omitting “sensitive” features such as race or sex, or obvious proxies for these. However, a model may leverage any feature associated with the outcome, and common measures of model performance and fairness will be essentially unaffected. Incidental correlated features may not be appropriate decision bases, or they may represent unfairness risks. Incidental feature risks are highest when appropriate predictors are not included in the model. Therefore, careful consideration of what might be missing is crucial.

Dataset

This article builds on results from a previous blog post and uses the same dataset and code base to illustrate the effects of missing and incidental features [1, 2]. In brief, I use a publicly-available loans dataset, in which the outcome is loan default status (binary), and predictors include income, employment length, debt load, etc. I preferentially (but randomly) sort lower-income cases into a made-up “female” category, and for simplicity consider only two gender categories (“males” and “females”). The result is that “females” on average have a lower income, but male and female incomes overlap; some females are high-income, and some males low-income. Examining common fairness and performance metrics, I found similar results whether the model relied on income or on gender to predict defaults, illustrating risks of relying only on metrics to detect bias.

My previous blog post showed what happens when an incidental feature substitutes for an appropriate feature. Here, I will discuss what happens when both the appropriate predictor and the incidental feature are included in the data. I test two model types, and show that, as might be expected, the female status contributes to predictions despite the fact it contains no additional information. However, the incidental feature contributes much more to the random forest model than to the XGBoost model, suggesting that model selection may be help reduce unfairness risk, although tradeoffs should be considered.

Fairness metrics and global importances

In my example, the female feature adds no information to a model that already contains income. Any reliance on female status is unnecessary and represents “direct discrimination” risk. Ideally, a machine learning algorithm would ignore such a feature in favor of the stronger predictor.

When the incidental feature, female status, is added to wither a random forest or XGBoost model, I see little change in overall performance characteristics or performance metrics (data not shown). ROC scores barely budge (as should be expected). False positive rates show very slight changes.

Demographic parity, or the difference in loan default rates for females vs. males, remain essentially unchanged for XGBoost (5.2% vs.5.3%) when the female indicator is included, but for random forest, this metric does change from 4.3% to 5.0%; I discuss this observation in detail below.

Global permutation importances show weak influences from the female feature for both model types. This feature ranks 12/14 for the random forest model, and 22/26 for XGBoost (when female=1). The fact that female status is of relatively low importance may seem reassuring, but any influence from this feature is a fairness risk.

There are no clear red flags in global metrics when female status is included in the data — but this is expected as fairness metrics are similar whether decisions are based on an incidental or causal factor [1]. The key question is: does incorporation of female status increase disparities in outcome?

Aggregated shapley values

We can measure the degree to which a feature contributes to differences in group predictions using aggregated Shapley values [3]. This technique distributes differences in predicted outcome rates across features so that we can determine what drives differences for females vs. males. Calculation involves constructing a reference dataset consisting of randomly selected males, calculating Shapley feature importances for randomly-selected females using this “foil”, and then aggregating the female Shapley values (also called “phi” values).

Results are shown below for both model types, with and without the “female” feature. The top 5 features for the model not including female is plotted along with female status for the model that includes that feature. All other features are summed into “other”.

Image by author

First, note that the blue bar for female (present for the model including female status only) is much larger for random forest than for XGBoost. The bar magnitudes indicate the amount of probability difference for women vs. men that is attributed to a feature. For random forest, the female status feature increases the probability of default for females relative to males by 1.6%, compared to 0.3% for XGBoost, an ~5x difference.

For random forest, female status ranks in the top 3 influential features in determining the difference in prediction for males vs. females, even though the feature was the 12th most important globally. The global importance does not capture this feature’s impact on fairness.

As mentioned in the section above, the random forest model shows decreased demographic parity when female status is included in the model. This effect is also apparent in the Shapley plots– the increase due to the female bar is not compensated for by any decrease in the other bars. For XGBoost, the small contribution from female status appears to be offset by tiny decreases in contributions from other features.

The reduced impact of the incidental feature for XGBoost compared to random forest makes sense when we think about how the algorithms work. Random forests create trees using random subsets of features, which are examined for optimal splits. Some of these initial feature sets will include the incidental feature but not the appropriate predictor, in which case incidental features may be chosen for splits. For XGBoost models, split criteria are based on improvements to a previous model. An incidental feature can’t improve a model based on a stronger predictor; therefore, after several rounds, we expect trees to include the appropriate predictor only.

Demographic parity decreases for random forest can also be understood considering model building mechanisms. When a subset of features to be considered for a split is generated in the random forest, we essentially have two “income” features, and so it’s more likely that (direct or indirect) income information will be selected.

The random forest model effectively uses a larger feature set than XGBoost. Although numerous features are likely to appear in both model types to some degree, XGBoost solutions will be weighted towards a smaller set of more predictive features. This reduces, but does not eliminate, risks related to incidental features for XGBoost.

Is XGBoost fairer than Random Forest?

In a previous blog post [4], I showed that incorporation of interactions to mitigate feature bias was more effective for XGBoost than for random forest (for one test scenario). Here, I observe that the XGBoost model is also less influenced by incidental information. Does this mean that we should prefer XGBoost for fairness reasons?

XGBoost has advantages when both an incidental and appropriate feature are included in the data but doesn’t reduce risk when only the incidental feature is included. A random forest model’s reliance on a larger set of features may be a benefit, especially when additional features are correlated with the missing predictor.

Furthermore, the fact that XGBoost doesn’t rely much on the incidental feature does not mean that it doesn’t contribute at all. It may be that only a smaller number of decisions are based on inappropriate information.

Leaving fairness aside, the fact that the random forest samples a larger portion of what you might think of as the “solution space”, and relies on more predictors, may be have some advantages for model robustness. When a model is deployed and faces unexpected errors in data, the random forest model may be somewhat more able to compensate. (On the other hand, if random forest incorporates a correlated feature that is affected by errors, it might be compromised while an XGBoost model remains unaffected).

XGBoost may have some fairness advantages, but the “fairest” model type is context-dependent, and robustness and accuracy must also be considered. I feel that fairness testing and explainability, as well thoughtful feature choices, are probably more valuable than model type in promoting fairness.

What am I missing?

Fairness considerations are crucial in feature selection for models that might affect lives. There are numerous existing feature selection methods, which generally optimize accuracy or predictive power, but do not consider fairness. One question that these don’t address is “what feature am I missing?”

A model that relies on an incidental feature that happens to be correlated with a strong predictor may appear to behave in a reasonable manner, despite making unfair decisions [1]. Therefore, it’s very important to ask yourself, “what’s missing?” when building a model. The answer to this question may involve subject matter expertise or additional research. Missing predictors thought to have causal effects may be especially important to consider [5, 6].

Obviously, the best solution for a missing predictor is to incorporate it. Sometimes, this may be impossible. Some effects can’t be measured or are unobtainable. But you and I both know that simple unavailability seldom determines the final feature set. Instead, it’s often, “that information is in a different database and I don’t know how to access it”, or “that source is owned by a different group and they are tough to work with”, or “we could get it, but there’s a license fee”. Feature choice generally reflects time and effort — which is often fine. Expediency is great when it’s possible. But when fairness is compromised by convenience, something does need to give. This is when fairness testing, aggregated Shapley plots, and subject matter expertise may be needed to make the case to do extra work or delay timelines in order to ensure appropriate decisions.

What am I including?

Another key question is “what am I including?”, which can often be restated as “for what could this be a proxy?” This question can be superficially applied to every feature in the dataset but should be very carefully considered for features identified as contributing to group differences; such features can be identified using aggregated Shapley plots or individual explanations. It may be useful to investigate whether such features contribute additional information above what’s available from other predictors

Who am I like, and what have they done before?

A binary classification model predicting something like loan defaults, likelihood to purchase a product, or success at a job, is essentially asking the question, “Who am I like, and what have they done before?” The word “like” here means similar values of the features included in the data, weighted according to their predictive contribution to the model. We then model (or approximate) what this cohort has done in the past to generate a probability score, which we believe is indicative of future results for people in that group.

The “who am I like?” question gets to the heart of worries that people will be judged if they eat too many pierogis, own too many cats, or just happen to be a certain race, sex, or ethnicity. The concern is that it is just not fair to evaluate individual people due to their membership in such groups, regardless of the average outcome for overall populations. What is appropriate depends heavily on context — perhaps pierogis are fine to consider in a heart attack model, but would be worrisome in a criminal justice setting.

Our models assign people to groups — even if models are continuous, we can think of that as the limit of very little buckets — and then we estimate risks for these populations. This isn’t much different than old-school actuarial tables, except that we may be using a very large feature set to determine group boundaries, and we may not be fully aware of the meaning of information we use in the process.

Final thoughts

Feature choice is more than a mathematical exercise, and likely requires the judgment of subject matter experts, compliance analysts, or even the public. A data scientist’s contribution to this process should involve using explainability techniques to populations and discover features driving group differences. We can also identify at-risk populations and ask questions about features known to have causal relationships with outcomes.

Legal and compliance departments often focus on included features, and their concerns may be primarily related to specific types of sensitive information. Considering what’s missing from a model is not very common. However, the question, “what’s missing?” is at least as important as, “what’s there?” in confirming that models make fair and appropriate decisions.

Data scientists can be scrappy and adept at producing models with limited or noisy data. There is something satisfying about getting a model that “works” from less than ideal information. It can be hard to admit that something can’t be done, but sometimes fairness dictates that what we have right now really isn’t enough — or isn’t enough yet.

Author: Valerie Carey

Source: Towards Data Science

Different Roles in Data Science

In this article, we will have a look at five distinct data careers, and hopefully provide some advice on how to get one's feet wet in this convoluted field.

The data-related career landscape can be confusing, not only to newcomers, but also to those who have spent time working within the field.

Get in where you fit in. Focusing on newcomers, however, I find from requests that I receive from those interested in join the data field in some capacity that there is often (and rightly) a general lack of understanding of what it is one needs to know in order to decide where it is that they fit in. In this article, we will have a look at five distinct data career archetypes, and hopefully provide some advice on how to get one's feet wet in this vast, convoluted field.

We will focus solely on industry roles, as opposed to those in research, as not to add an additional layer of complication. We will also omit executive level positions such as Chief Data Officer and the like, mostly because if you are at the point in your career that this role is an option for you, you probably don't need the information in this article.

So here are 5 data career archetypes, replete with descriptions and information on what makes them distinct from one another.

Source: KDnuggets

Data Architect

The data architect focuses on engineering and managing data stores and the data that reside within them.

The data architect is concerned with managing data and engineering the infrastructure which stores and supports this data. There is generally little to no data analysis needing to take place in such a role (beyond data store analysis for performance tuning), and the use of languages such as Python and R is likely not necessary. An expert level knowledge of relational and non-relational databases, however, will undoubtedly be necessary for such a role. Selecting data stores for the appropriate types of data being stored, as well as transforming and loading the data, will be necessary. Databases, data warehouses, and data lakes; these are among the storage landscapes that will be in the data architect's wheelhouse. This role is likely the one which will have the greatest understanding of and closest relationship with hardware, primarily that related to storage, and will probably have the best understanding of cloud computing architectures of anyone else in this article as well.

SQL and other data query languages — such as Jaql, Hive, Pig, etc. — will be invaluable, and will likely be some of the main tools of an ongoing data architect's daily work after a data infrastructure has been designed and implemented. Verifying the consistency of this data as well as optimizing access to it are also important tasks for this role. A data architect will have the know-how to maintain appropriate data access rights, ensure the infrastructure's stability, and guarantee the availability of the housed data.

This is differentiated from the data engineer role by focus: while a data engineer is concerned with building and maintaining data pipelines (see below), the data architect is focused on the data itself. There may be overlap between the 2 roles, however: ETL; any task which could transform or move data, especially from one store to another; starting data on a journey down a pipeline.

Like other roles in this article, you might not necessarily see a "data architect" role advertised as such, and might instead see related job titles, such as:

• Database Administrator
• Spark Administrator
• Big Data Administrator
• Database Engineer
• Data Manager

Data Engineer

The data engineer focuses on engineering and managing the infrastructure which supports the data and data pipelines.

What is the data infrastructure? It's the collection of software and storage solutions that allow for the retrieval of data from a data store, the processing of data in some specified manner (or series of manners), the movement of data between tasks (as well as the tasks themselves), as data is on its way to analysis or modeling, as well as the tasks which come after this analysis or modeling. It's the pathway that the data takes as it moves along its journey from its home to its ultimate location of usefulness, and beyond. The data engineer is certainly familiar with DataOps and its integration into the data lifecycle.

From where does the data infrastructure come? Well, it needs to be designed and implemented, and the data engineer does this. If the data architect is the automobile mechanic, keeping the car running optimally, then data engineering can be thought of as designing the roadway and service centers that the automobile requires to both get around and to make the changes needed to continue on the next section of its journey. The pair of these roles are crucial to both the functioning and movement of your automobile, and are of equal importance when you are driving from point A to point B.

Truth be told, some the technologies and skills required for data engineering and data management are similar; however, the practitioners of these disciplines use and understand these concepts at different levels. The data engineer may have a foundational knowledge of securing data access in a relational database, while the data architect has expert level knowledge; the data architect may have some understanding of the transformation process that an organization requires its stored data to undergo prior to a data scientist performing modeling with that data, while a data engineer knows this transformation process intimately. These roles speak their own languages, but these languages are more or less mutually intelligible.

You might find related job titles advertised for such as:

• Big Data Engineer
• Data Infrastructure Engineer

Data Analyst

The data analyst focuses on the analysis and presentation of data.

I'm using data analyst in this context to refer to roles related strictly to the descriptive statistical analysis and presentation of data. This includes the preparation of reporting, dashboards, KPIs, business performance metrics, as well as encompassing anything referred to as "business intelligence." The role often requires interaction with (or querying of) databases, both relational and non-relational, as well as with other data frameworks.

While the previous pair of roles were related to designing the infrastructure to manage and facilitate the movement of the data, as well managing the data itself, data analysts are chiefly concerned with pulling from the data and working with it as it currently exists. This can be contrasted with the following 2 roles, machine learning engineers and data scientists, both of which focus on eliciting insights from data above and beyond what it already tells us at face value. If we can draw parallels between data scientists and inferential statisticians, then data analysts are descriptive statisticians; here is the current data, here is what it looks like, and here is what we know from it.

Data analysts require a unique set of skills among the roles presented. Data analysts need to have an understanding of a variety of different technologies, including SQL & relational databases, NoSQL databases, data warehousing, and commercial and open-source reporting and dashboard packages. Along with having an understanding of some of the aforementioned technologies, just as important is an understanding of the limitations of these technologies. Given that a data analyst's reporting can often be ad hoc in nature, knowing what can and cannot be done without spending an ordination amount of time on a task prior to coming to this determination is important. If an analyst knows how data is stored, and how it can be accessed, they can also know what kinds of requests — often from people with absolutely no understanding of this — are and are not serviceable, and can suggest ways in which data can be pulled in a useful manner. Knowing how to quickly adapt can be key for a data analyst, and can separate the good from the great.

Related job titles include:

• Business Analyst
• Business Intelligence (BI) Analyst

Machine Learning Engineer

The machine learning engineer develops and optimizes machine learning algorithms, and implements and manages (near) production level machine learning models.

Machine learning engineers are those crafting and using the predictive and correlative tools used to leverage data. Machine learning algorithms allow for the application of statistical analysis at high speeds, and those who wield these algorithms are not content with letting the data speak for itself in its current form. Interrogation of the data is the modus operandi of the machine learning engineer, but with enough of a statistical understanding to know when one has pushed too far, and when the answers provided are not to be trusted.

Statistics and programming are some of the biggest assets to the machine learning researcher and practitioner. Maths such as linear algebra and intermediate calculus are useful for those employing more complex algorithms and techniques, such as neural networks, or working in computer vision, while an understanding of learning theory is also useful. And, of course, a machine learning engineer must have an understanding of the inner workings of an arsenal of machine learning algorithms (the more algorithms the better, and the deeper the understanding the better!).

Once a machine learning model is good enough for production, a machine learning engineer may also be required to take it to production. Those machine learning engineers looking to do so will need to have knowledge of MLOps, a formalized approach for dealing with the issues arising in productionizing machine learning models.

Related job titles:

• Machine Learning Scientist
• Machine Learning Practitioner
• <specific machine learning technology> Engineer, e.g. Natural Language Processing Engineer, Computer Vision Engineer, etc.

Data Scientist

The data scientist is concerned primarily with the data, the insights which can be extracted from it, and the stories that it can tell.

The data architect and data engineer are concerned with the infrastructure which houses and transports the data. The data analyst is concerned with pulling descriptive facts from the data as it exists. The machine learning engineer is concerned with advancing and employing the tools available to leverage data for predictive and correlative capabilities, as well as making the resulting models widely-available. The data scientist is concerned primarily with the data, the insights which can be extracted from it, and the stories that it can tell, regardless of what technologies or tools are needed to carry out that task.

The data scientist may use any of the technologies listed in any of the roles above, depending on their exact role. And this is one of the biggest problems related to "data science"; the term means nothing specific, but everything in general. This role is the Jack Of All Trades of the data world, knowing (perhaps) how to get a Spark ecosystem up and running; how to execute queries against the data stored within; how to extract data and house in a non-relational database; how to take that non-relational data and extract it to a flat file; how to wrangle that data in R or Python; how to engineer features after some initial exploratory descriptive analysis; how to select an appropriate machine learning algorithm to perform some predictive analytics on the data; how to statistically analyze the results of said predictive task; how to visualize the results for easy consumption by non-technical folks; and how to tell a compelling story to executives with the end result of the data processing pipeline just described.

And this is but one possible set of skills a data scientist may possess. Regardless, however, the emphasis in this role is on the data, and what can be gleaned from it. Domain knowledge is often a very large component of such a role as well, which is obviously not something that can be taught here. Key technologies and skills for a data scientist to focus on are statistics (!!!), programming languages (particularly Python, R, and SQL), data visualization, and communication skills — along with everything else noted in the above archetypes.

There can be a lot of overlap between the data scientist and the machine learning engineer, at least in the realm of data modeling and everything that comes along with that. However, there is often confusion as to what the differences are between these roles as well. For a very solid discussion of the relationship between data engineers and data scientists, a pair of roles which also can also have significant overlap, have a look at this great article by Mihail Eric.

Remember that these are simply archetypes of five major data profession roles, and these can vary between organizations. The flowchart in the image from the beginning of the article can be useful in helping you navigate the landscape and where you might find your role within it. Enjoy the ride to your ideal data profession!

Author: Matthew Mayo

Source: KDnuggets

While decision makers are well aware that digital transformation is essential to their organizations’ future, many are jumping into new technologies that don’t align with their current digital transformation pain points, according to a new report from PointSource, a division of Globant that provides IT solutions.

All too often decision makers invest in technologies without taking a step back to assess how those technologies fit into their larger digital strategy and business goals, the study said. While the majority of such companies perceive these investments as a fast track to the next level of digital maturity, they are actually taking an avoidable detour. 

PointSource surveyed more than 600 senior-level decision makers and found that a majority are investing in technology that they don’t feel confident using. In fact, at least a quarter plan to invest more than 25 percent of their 2018 budgets in artificial intelligence (AI), blockchain, voice-activated technologies or facial-recognition technologies.

However, more than half (53 percent) of companies do not feel prepared to effectively use AI, blockchain or facial-recognition technologies.

See Also A look inside American Family Insurance's digital transformation office

Companies are actively focusing on digital transformation, the survey showed. Ninety-four percent have increased focus on digital growth within the last year, and 90 percent said digital plays a central role in their overarching business goals.
Fifty-seven percent of senior managers are unsatisfied with one or more of the technologies their organizations’ employees rely on. 

Many companies feel digitally outdated, with 45 percent of decision makers considering their company’s digital infrastructure to be outdated compared with that of their competitors.

Author: Bob Violino

Source: Information Management

What distinguishes strong from weak leaders? This raises the question if leaders are born or can be grown. It is the classic “nature versus nurture” debate. What matters more? Genes or your environment?

This question got me to thinking about whether data scientists and business analysts within an organization can be more than just a support to others. Can they be become leaders similar to C-level executives? 

Three primary success factors for effective leaders

Having knowledge means nothing without having the right types of people. One person can make a big difference. They can be someone who somehow gets it altogether and changes the fabric of an organization’s culture not through mandating change but by engaging and motivating others.

For weak and ineffective leaders irritating people is not only a sport for them but it is their personal entertainment. They are rarely successful. 

You know there is a problem when a leader says, “I don’t do that; I have people who do that.” Good leaders do not necessarily have high intelligence, good memories, deep experience, or innate abilities that they are born with. They have problem solving skills. 

As an example, the Ford Motor Company’s CEO Alan Mulally came to the automotive business from Boeing in the aerospace industry. He was without deep automotive industry experience. He has been successful at Ford. Why? Because he is an analytical type of leader.

Effective managers are analytical leaders who are adaptable and possess systematic and methodological ways to achieve results. It may sound corny but they apply the “scientific method” that involves formulating hypothesis and testing to prove or disprove them. We are back to basics.

A major contributor to the “scientific method” was the German mathematician and astronomer Johannes Kepler. In the early 1600s Kepler’s three laws of planetary motion led to the Scientific Revolution. His three laws made the complex simple and understandable, suggesting that the seemingly inexplicable universe is ultimately lawful and within the grasp of the human mind. 

Kepler did what analytical leaders do. They rely on searching for root causes and understanding cause-and-effect logic chains. Ultimately a well-formulated strategy, talented people, and the ability to execute the executive team’s strategy through robust communications are the key to performance improvement. 

Key characteristics of the data scientist or analyst as leader

The popular Moneyball book and subsequent movie about baseball in the US demonstrated that traditional baseball scouts methods (e.g., “He’s got a good swing.”) gave way to fact-based evidence and statistical analysis. Commonly accepted traits of a leader, such as being charismatic or strong, may also be misleading.

My belief is that the most scarce resource in an organization is human ability and competence. That is why organizations should desire that every employee be developed for growth in their skills. But having sound competencies is not enough. Key personal qualities complete the package of an effective leader. 

For a data scientist or analyst to evolve as an effective leader three personal quality characteristics are needed: curiosity, imagination, and creativity. The three are sequentially linked. Curious people constantly ask “Why are things the way they are?” and “Is there a better way of doing things?” Without these personal qualities then innovation will be stifled. The emergence of analytics is creating opportunities for analysts as leaders. 

Weak leaders are prone to a diagnostic bias. They can be blind to evidence and somehow believe their intuition, instincts, and gut-feel are acceptable masquerades for having fact-based information. In contrast, a curious person always asks questions. They typically love what they do. If they are also a good leader they infect others with enthusiasm. Their curiosity leads to imagination. Imagination considers alternative possibilities and solutions. Imagination in turn sparks creativity.

Creativity is the implementation of imagination

Good data scientists and analysts have a primary mission: to gain insights relying on quantitative techniques to result in better decisions and actions. Their imagination that leads to creativity can also result in vision. Vision is a mark of a good leader. In my mind, an executive leader has one job (aside from hiring good employees and growing them). That job is to answer the question, “Where do we want to go?” 

After that question is answered then managers and analysts, ideally supported by the CFO’s accounting and finance team, can answer the follow-up question, “How are we going to get there?” That is where analytics are applied with the various enterprise and corporate performance management (EPM/CPM) methods that I regularly write about. EPM/CPM methods include a strategy map and its associated balance scorecard with KPIs; customer profitability analysis; enterprise risk management (ERM), and capacity-sensitive driver-based rolling financial forecasts and plans. Collectively they assure that the executive team’s strategy can be fully executed.

My belief is that that other perceived characteristics of a good leader are over-rated. These include ambition, team spirit, collegiality, integrity, courage, tenacity, discipline, and confidence. They are nice-to-have characteristics, but they pale compared to the technical competency and critical thinking and communications skills that I earlier described. 

Be analytical and you can be a leader. You can eventually serve in a C-suite role

Author: Gary Cokins 

Source: Information Management

Do We Need Decision Scientists?

Twenty years ago, there was a great reckoning in the market research industry. After spending decades mastering the collection and analysis of survey data, the banality of research-backed statements like “consumers don’t like unhealthy products” belied the promise of consumer understanding. Instead of actionable insights, business leaders received detailed reports filled with charts and tables providing statistically proven support for research findings that did little to help decision-makers figure out what to do.

So, the market research industry transformed itself into an insight industry over the past twenty years. To meet the promise of consumer understanding, market researchers are now focusing on applying business knowledge and synthesizing data to drive better insights and ultimately better decisions.

A similar reckoning is at hand for data scientists and the AI models they create. Data scientists are mathematical programmer geeks that can work near-miracles with massive complex data sets. But their focus on complex data is much like the old market research focus on surveying consumers – without a clear focus on applying business knowledge to support decision-making, the results are often as banal as the old market research reports. Improving data and text mining techniques and renaming them Machine Learning, then piling on more data and calling it Big Data, doesn’t change that fundamental problem.

Today’s data scientists can learn a lot from companies like Johnson & Johnson, Colgate, and Bayer. These leaders have successfully transformed their market research functions into insight generators and decision enablers by combining analytical tools with the business skills required to drive better decisions.

Data scientists could follow a similar path, but what if we took a much bigger leap?

Leaping to Decision Scientists

According to Merriam-Webster, science is “knowledge about or study of the natural world based on facts learned through experiments and observation.”

Applying that definition to data science highlights the critical disconnect. Data scientists do not exist inside companies to study data – they are there to generate knowledge to help business decision-makers make better decisions. By itself, collecting more data and analyzing it faster does not result in objective knowledge that can be relied on for better decision-making. The science of data is not the science of business.

Now imagine the evolution of a new decision scientist role. Since decision-making effectiveness is almost perfectly correlated with business performance">almost perfectly correlated with business performance, a focus on studying and building knowledge about business decisions and decision-making will directly advance business goals.

Rather than studying how to generate, process and analyze generalized business data, tomorrow’s decision scientists will focus on tracking, understanding and improving business decision-making.

Decision scientists will live in an exciting new world where the current lack of scientific knowledge about business decision-making means major discoveries will happen every day. Even better, every decision is a natural experiment, setting the stage for incredibly rapid learning. Finally, businesses will benefit directly by applying new understanding to make better, faster decisions.

Decision scientists will focus on mapping the structure of business decisions and understanding the process business people use to make decisions, including the mix of data, experience and intuition needed to guide recommendations and decisions that deliver business value. 

Decision Scientists Will Decode the DNA of Business

By mapping the decisions that drive a business and then tracking those decisions’ inputs and outputs, decision scientists can bring decision-centric, business-focused scientific direction to the disconnected layers of today’s data-centric world. Much like a complete map of our DNA highlights the genes and interventions important for better health, a comprehensive map of our decisions can focus efforts to drive the most business impact.

• Decisions – Who makes decisions, and how are their choices made? How can we measure decision success and map that back to the optimal mix of decision inputs and most effective decision processes?
• Business Issues – How can decisions be used to model a predictable connection between the inputs and outputs of a business? How are business goals connected, via decisions, from the CEO to the front-line manager?
• Insights – What is the scientific definition of an insight? What is the best role for human synthesis in generating insights, and can that human role be modeled and even automated for faster, more efficient insight generation?
• Analytics – What analysis and mathematical models are needed to support critical decisions? How can we align top-down analysis that starts with key decisions and business issues with bottom-up analysis that starts with key variables and data sources?

Decision scientists will shift the focus from the science of inputs (data and analysis) to the science of outputs (recommendations and decisions). Of course, data science will continue as an important activity, except now it will be directed not only by the technical challenges of complex data sets but also by the complex needs of decision-makers. This shift will significantly improve the business value of data and analytics, making tomorrow’s decision scientists an indispensable business resource.

Author: Erik Larson

Source: Forbes

E-commerce and the growing importance of data

E-commerce is claiming a bigger role in global retail. In the US for example, e-commerce currently accounts for approximately 10% of all retail sales, a number that is projected to increase to nearly 18% by 2021. To a large extent, the e-commerce of the present exists in the shadow of the industry’s early entrant and top player, Amazon. Financial analysts predict that the retail giant will control 50% of the US’ online retail sales by 2021, leaving other e-commerce stores frantically trying to take a page out of the company’s incredibly successful online retail playbook.

While it seems unlikely that another mega-retailer will rise to challenge Amazon’s e-commerce business in the near future, at least 50% of the online retail market is wide open. Smaller and niche e-commerce stores have a ;arge opportunity to reach specialized audiences, create return customers, and cultivate persistent brand loyalty. Amazon may have had a first-mover advantage, but the rise in big data and the ease of access to analytics means that smaller companies can find areas in which to compete and improve margins. As e-retailers look for ways to expand revenues while remaining lean, data offers a way forward for smart scalability.

Upend your back-end

While data can improve e-commerce’s customer-facing interactions, it can have just as major an impact on the customer experience factors that take place off camera. Designing products that customers want, having products in stock, making sure that products ship on schedule, all these kind of back-end operations play a part in shaping customer experience and satisfaction. In order to shift e-commerce from a product-centric to a consumer-centric model, e-commerce companies need to invest in unifying customer data to inform internal processes and provide faster, smarter services.

The field of drop shipping, for instance, is coming into its own thanks to smart data applications. Platforms like Oberlo are leveraging prescriptive analytics to enable intelligent product selection for Shopify stores, helping them curate trending inventory that sells, allowing almost anyone to create their own e-store. Just as every customer touchpoint can be enhanced with big data, e-commerce companies that apply unified big data solutions to their behind-the-scenes benefit from streamlined processes and workflow.

Moreover, e-commerce companies that harmonize data across departments can identify purchasing trends and act on real-time data to optimize inventory processes. Using centralized data warehouse software like Snowflake empowers companies to create a single version of customer truth to automate reordering points and determine what items they should be stocking in the future. Other factors, such as pricing decisions, can also be finessed using big data to generate specific prices per product that match customer expectations and subsequently sell better.

Data transforms the customer experience

When it comes to how data can impact the overall customer experience, e-commerce companies don’t have to invent the wheel. There’s a plethora of research that novice and veteran data explorers can draw on when it comes to optimizing customer experiences on their websites. General findings on the time it takes for customers to form an opinion of a website, customers’ mobile experience expectations, best times to send promotional emails and many more metrics can guide designers and developers tasked with improving e-commerce site traffic.

However, e-commerce sites that are interested in more benefits will need to invest in more specific data tools that provide a 360-degree view of their customers. Prescriptive analytic tools like Tableau empower teams to connect the customer dots by synthesizing data across devices and platforms. Data becomes valuable as it provides insights that allow companies to make smarter decisions based on each consumer identify inbound marketing opportunities and automate recommendations and discounts based on the customer’s previous behavior.

Data can also inspire changes in a field that has always dominated the customer experience: customer support. The digital revolution has brought substantial changes in the once sleepy field of customer service, pioneering new ways of direct communication with agents via social media and introducing the now ubiquitous AI chatbots. In order to provide the highest levels of customer satisfaction throughout these new initiatives, customer support can utilize data to anticipate when they might need more human agents staffing social media channels or the type of AI persona that their customers want to deal with. By improving customer service with data, e-commerce companies can improve the entire customer experience.

Grow with your data

As more and more data services migrate to the cloud, e-commerce companies have ever-expanding access to flexible data solutions that both fuel growth and scale alongside the businesses they’re helping. Without physical stores to facilitate face-to-face relationships, e-commerce companies are tasked with transforming their digital stores into online spaces that customers connect with and ultimately want to purchase from again and again.

Data holds the key to this revolution. Instead of trying to force their agenda upon customers or engage in wild speculations about customer desires, e-commerce stores can use data to craft narratives that engage customers, create a loyal brand following, and drive increasing profits. With only about 2.5% of e-commerce & web visits converting to saleson average, e-commercecompanies that want to stay competitive must open themselves up to big data and the growth opportunities it offers.

Author: Ralph Tkatchuck

Source: Dataconomy

Edge computing in a Nutshell

Edge computing (EC) allows data generated by the Internet of Things (IoT) to be processed near its source, rather than sending the data great distances, to data centers or a cloud. More specifically, edge computing uses a network of micro-data stations to process or store the data locally, within a range of 100 square feet. Prior to edge computing, it was assumed all data would be sent to the cloud using a large and stable pipeline between the edge/IoT device and the cloud.

Typically, IoT devices transfer data, sometimes massive amounts, sending it all to a data center, or cloud, for processing. With edge computing, processing starts near the source. Once the initial processing has occurred, only the data needing further analysis is sent. EC screens the data locally, reducing the volume of data traffic sent to the central repository.

This tactic allows organizations to process data in “almost” real time. It also reduces the network’s data stream volume and eliminates the potential for bottlenecks. Additionally, nearby edge devices can “potentially” record the same information, providing backup data for the system.

A variety of factors are promoting the expansion of edge computing. The cost of sensors has been decreasing, while simultaneously, the pace of business continues to increase, with real-time responses providing a competitive advantage to its users. Businesses using edge computing can analyze and store portions of data quickly and inexpensively. Some are theorizing edge computing means an end to the cloud. Others believe it will complement and support cloud computing.

The Uses of Edge Computing

Edge computing can be used to help resolve a variety of situations. When IoT devices have a poor connectivity, or when the connection is intermittent, edge computing provides a convenient solution because it doesn’t need a connection to process the data, or make a decision.

It also has the effect of reducing time loss, because the data doesn’t have to travel across a network to reach a data center or cloud. In situations where a loss of milliseconds is unacceptable, such as in manufacturing or financial services, edge computing can be quite useful.

Smart cities, smart buildings, and building management systems are ideal for the use of edge computing. Sensors can make decisions on the spot, without waiting for a decision from another location. Edge computing can be used for energy and power management, controlling lighting, HVAC, and energy efficiency.

A few years ago, PointGrab announced an investment in CogniPointTM, and its Edge Analytics sensor solution for smart buildings, by Philips Lighting and  Mitsubishi UFJ Capital. PointGrab is a company which provides smart sensor solutions to automated buildings.

The company uses a deep learning technology in developing its sensors, which detects the occupant’s locations, maintains a head count, monitors their movements, and adjusts its internal environment using real-time analytics. PointGrab’s Chief Business Officer, Itamar Rothat stated:

“CogniPoint’s ultra-intelligent edge-analytics sensor technology will be a key facilitator for capturing critical data for building operations optimization, energy savings improvement, and business intelligence.”

Another example of edge computing is the telecommunication companies’ expansion of 5G cellular networks. Kelly Quinn, an IDC research manager, predicts telecom providers will add micro-data stations that are integrated into 5G towers, or located near the towers. Business customers can own or rent the micro-data stations for edge computing. (If rented, negotiate direct access to the provider’s broader network, which can then connect to an in-house data center, or cloud.)

Edge Computing vs. Fog Computing

Edge computing and fog computing both deal with processing and screening data prior to its arrival at a data center or cloud. Technically, edge computing is a subdivision of fog computing. The primary difference is where the processing takes place.

With fog computing, the processing typically happens near the local area network (but technically, can happen anywhere between the edge and a data center/cloud), using a fog node or an IoT gateway to screen and process data. Edge computing processes data within the same device, or a nearby one, and uses the communication capabilities of edge gateways or appliances to send the data. (A gateway is a device/node that opens and closes to send and receive data. A gateway node can be part of a network’s “edge.”)

Edge Computing Security

There are two arguments regarding the security of edge computing. Some suggest security is better with edge computing because the data stays closer to its source and does not move through a network. They argue the less data stored in a corporate data center, or cloud, the less data that is vulnerable to hackers.

Others suggest edge computing is significantly less secure because “edge devices” can be extremely vulnerable, and the more entrances to a system, the more points of attack available to a hacker. This makes security an important aspect in the design of any “edge” deployment. Access control, data encryption, and the use of virtual private network tunneling are important parts of defending an edge computing system.

The Need for Edge Computing

There is an ever-increasing number of sensors providing a base of information for the Internet of Things. It has traditionally been a source of big data. Edge computing, however, attempts to screen the incoming information, processing useful data on the spot, and sending it directly to the user. Consider the sheer volume of data being supplied to the Internet of Things by airports, cities, the oil drilling industry, and the smart phone industry. The huge amounts of data being communicated creates problems with network latency, bandwidth, and the most significant problem, speed. Many IoT applications are mission-critical, and the need for speed is crucial.

EC can lower costs and provide a smooth flow of service. Mission critical data can be analyzed, allowing a business to choose the services running at the edge, and to screen data sent to the cloud, lowering IoT costs and getting the most value from IoT data transfers. Additionally, edge computing provides “Screened” big data.

Transmitting immense amounts of data is expensive and can strain a network’s resources. Edge computing processes data from, or near, the source, and sends only relevant data through network to a data processor or cloud. For instance, a smart refrigerator doesn’t need to continuously send temperature data to a cloud for analysis. Instead, the refrigerator can be designed to send data only when the temperature changes beyond a certain range, minimizing unnecessary data. Similarly, a security camera would only send data after detecting motion.

Depending on how the system is designed, edge computing can direct manufacturing equipment (or other smart devices) to continue operating without interruption, should internet connectivity become intermittent, or drop off, completely, providing an ideal backup system.

It is an excellent solution for businesses needing to analyze data quickly in unusual circumstances, such as airplanes, ships, and some rural areas. For example, edge devices could detect equipment failures, while “not” being connected to a cloud or control system. Examples of edge computing include:

Internet of Things

• Smart streetlights
• Home appliances
• Motor vehicles (Cars and trucks)
• Traffic lights
• Thermostats
• Mobile devices

Industrial Internet of Things (IIoT)

• Smart power grid technology
• Magnetic resonance (MR) scanner
• Automated industrial machines
• Undersea blowout preventers
• Wind turbines

Edge Computing Compliments the Cloud

The majority of businesses using EC continue to use the cloud for data analysis. They use a combination of the systems, depending on the problem. In some situations, the data is processed locally, and in others, data is sent to the cloud for further analysis. The cloud can manage and configure IoT devices, and analyze the “Screened” big data provided by Edge Devices. Combining the power of edge computing and the cloud maximizes the value of Internet of Things. Businesses will have the ability to analyze screened big data, and act on it with greater speed and precision, offering an advantage against competitors.

Data Relationship Management

Device Relationship Management (DRM) is about monitoring and maintaining equipment using the Internet, and includes controlling these “sensors on the edge.” DRM is designed specifically to communicate with the software and microprocessors of IoT devices and lets organizations supervise and schedule the maintenance of its devices, ranging from printers to industrial machines to data storage systems. DRM provides preventative maintenance support by giving organizations detailed diagnostic reports, etc. If an edge device is lacking the necessary hardware or software, these can be installed. Outsourcing maintenance on edge devices can be more cost effective at this time than hiring an in-house maintenance staff, particularly if the maintenance company can access the system by way of the internet.

Author: Keith D. Foote

Source: Dataversity

Een eerste indruk van de fusie tussen Cloudera en Hortonworks

Een aantal maanden geleden werd bekend dat big data-bedrijven Cloudera en Hortonworks gaan fuseren. De overname is inmiddels goedgekeurd en Cloudera en Hortonworks gaan verder als één bedrijf. Techzine ging in gesprek met Wim Stoop, senior product marketing manager bij Cloudera. Stoop heeft alle ins en outs wat betreft de visie rond deze fusie en wat de fusie betekent voor bedrijven en data analisten die met de producten van de twee bedrijven werken.

Stoop vertelt dat deze fusie min of meer het perfecte huwelijk is. Beide bedrijven houden zich bezig met big data op basis van Hadoop en hebben zich de afgelopen jaren hierin gespecialiseerd. Zo is Hortonworks erg goed in Hadoop Data Flow (HDF), werken met streaming data die snel in het Hadoop platform moeten worden toegevoegd. 

Cloudera data science workbench

Cloudera heeft met zijn data science workbench een goede oplossing in handen voor data analisten. Zij kunnen met deze workbench snel en eenvoudig data combineren en analyseren, zonder dat je daarvoor direct extreem veel rekenkracht nodig hebt. Met de workbench van Cloudera kun je experimenteren en testen om te zien wat voor uitkomsten dit biedt, voordat je het meteen op grote schaal toepast. Het belangrijkste voordeel is dat de workbench overweg kan met enorm veel programmeertalen, waardoor iedere data analist in zijn eigen favoriete taal kan werken. De workbench houdt tevens exact bij welke stappen zijn doorlopen om tot een resultaat te komen. De uitkomst is weliswaar belangrijk, maar het algoritme en methoden die leiden tot het eindresultaat zijn minstens net zo belangrijk.

De route naar één oplossing

Als je er dieper op in gaat dan zijn er natuurlijk veel meer zaken waar juist Hortonworks of Cloudera heel erg goed in is. Of welke technologie net even beter of efficiënter is dan de andere. Dat zal het nieuwe bedrijf dwingen tot harde keuzes, maar volgens Stoop gaat dat allemaal wel goed komen. De behoefte aan een goed dataplatform is enorm groot, dat er dan keuzes gemaakt moeten worden is onvermijdelijk. Uiteindelijk speelt het bedrijf hiermee in op de kritiek die er op Hadoop is. Hadoop zelf vormt de basis van de database, maar daarboven zijn er zo veel verschillende modules die data kunnen inlezen, uitlezen of verwerken. Daardoor is het overzicht soms ver te zoeken. Het feit dat er zoveel oplossingen zijn heeft te maken met het open source karakter en de steun van bedrijven als Cloudera en Hortonworks, die bij veel projecten de grootste bijdrager zijn. Dat gaat ook veranderen met deze fusie. Er komt dit jaar nog een nieuw platform met de naam Cloudera Data Platform. In dit platform zullen de beste onderdelen van Hortonworks en Cloudera worden samengevoegd. Het betekent ook dat conflicterende projecten of modules goed nieuws zullen zijn voor de een maar slecht nieuws voor de ander. Voor het verwerken van metadata gebruiken beide bedrijven nu een andere oplossing, in het Cloudera Data Platform zullen we er maar één terug zien. Dat betekent dat het aantal modules een stukje minder wordt en alles overzichtelijker wordt, wat voor alle betrokkenen positief is.

Cloudera Data Platform

De naam van het nieuwe bedrijf was nog niet aan bod gekomen. De bedrijven hebben gekozen voor een fusie, maar uiteindelijk zal de naam Hortonworks gewoon verdwijnen. Het bedrijf gaat verder als Cloudera, vandaar ook de naam Cloudera Data Platform. De bedoeling is dat het Cloudera Data Platform dit jaar nog beschikbaar wordt, zodat klanten ermee kunnen gaan testen. Zodra het platform stabiel en volwassen genoeg is, krijgen klanten het advies om te migreren naar dit nieuwe platform. Alle bestaande Cloudera en Hortonworks producten zullen uiteindelijk gaan verdwijnen, maar tot eind 2022 blijven deze producten wel volledig ondersteund. Daarna moet iedereen echter over op het Cloudera Data Platform. Cloudera heeft in de meest recente versies van zijn huidige producten al rekening gehouden met een migratietraject. Bij Hortonworks zal dit vanaf nu ook gaan gebeuren. Het bedrijf gaat stappen zetten zodat bestaande producten en het nieuwe Data Platform in staat zijn om samen te werken bij de migratie naar het nieuwe platform.

Shared data experience

Een andere innovatie die volgens Stoop in de toekomst steeds belangrijker wordt is de shared data experience. Als klanten Cloudera producten gebruiken dan kunnen deze Hadoop-omgevingen eenvoudig aan elkaar gekoppeld worden, zodat ook de resources (CPU, GPU, geheugen) gecombineerd kunnen worden bij het analyseren van data. Stel dat een bedrijf Cloudera-omgevingen voor data-analyses heeft in eigen datacenters én cloudplatformen, maar dat het daarna ineens een heel groot project moet analyseren. In dat geval zou het al die omgevingen kunnen combineren en gezamenlijk kunnen inzetten. Daarnaast is het mogelijk om bijvoorbeeld data van lokale kantoren/filialen te combineren.

Door fusie meer innovatie mogelijk

Een gigantisch voordeel van deze fusie is volgens Stoop de ontwikkelcapaciteit die beschikbaar wordt om nieuwe innovatieve oplossingen te ontwikkelen. De bedrijven waren nu vaak afzonderlijk van elkaar aan vergelijkbare projecten aan het werken. Beide bedrijven droegen bijvoorbeeld bij aan een verschillend project dat om kan gaan met metadata in Hadoop. Uiteindelijk was een van de twee het wiel opnieuw aan het uitvinden, dat is nu niet meer nodig. Gezien de huidige arbeidsmarkt is het vinden van ontwikkelaars die de juiste passie en kennis hebben voor data analyse enorm lastig. Met deze fusie kan er veel efficiënter gewerkt gaan worden en kunnen er flink wat teams ingezet worden voor het ontwikkelen van nieuwe innovatieve oplossingen. Deze week vindt de Hortonworks Datasummit plaats in Barcelona. Daar zal ongetwijfeld meer bekend worden gemaakt over de fusie, de producten en de status van het nieuwe Cloudera Data Platform.

Auteur: Coen van Eenbergen

Bron: Techzine

Effective data analysis methods in 10 steps

In this data-rich age, understanding how to analyze and extract true meaning from the digital insights available to our business is one of the primary drivers of success.

Despite the colossal volume of data we create every day, a mere 0.5% is actually analyzed and used for data discovery, improvement, and intelligence. While that may not seem like much, considering the amount of digital information we have at our fingertips, half a percent still accounts for a huge amount of data.

With so much data and so little time, knowing how to collect, curate, organize, and make sense of all of this potentially business-boosting information can be a minefield, but online data analysisis the solution.

To help you understand the potential of analysis and how you can use it to enhance your business practices, we will answer a host of important analytical questions. Not only will we explore data analysis methods and techniques, but we’ll also look at different types of data analysis while demonstrating how to do data analysis in the real world with a 10-step blueprint for success.

What is a data analysis method?

Data analysis methods focus on strategic approaches to taking raw data, mining for insights that are relevant to a business’s primary goals, and drilling down into this information to transform metrics, facts, and figures into initiatives that benefit improvement.

There are various methods for data analysis, largely based on two core areas: quantitative data analysis methods and data analysis methods in qualitative research.

Gaining a better understanding of different data analysis techniques and methods, in quantitative research as well as qualitative insights, will give your information analyzing efforts a more clearly defined direction, so it’s worth taking the time to allow this particular knowledge to sink in.

Now that we’ve answered the question, ‘what is data analysis?’, considered the different types of data analysis methods, it’s time to dig deeper into how to do data analysis by working through these 10 essential elements.

1. Collaborate your needs

Before you begin to analyze your data or drill down into any analysis techniques, it’s crucial to sit down collaboratively with all key stakeholders within your organization, decide on your primary campaign or strategic goals, and gain a fundamental understanding of the types of insights that will best benefit your progress or provide you with the level of vision you need to evolve your organization.

2. Establish your questions

Once you’ve outlined your core objectives, you should consider which questions will need answering to help you achieve your mission. This is one of the most important steps in data analytics as it will shape the very foundations of your success.

To help you ask the right things and ensure your data works for you, you have to ask the right data analysis questions.

3. Harvest your data

After giving your data analytics methodology real direction and knowing which questions need answering to extract optimum value from the information available to your organization, you should decide on your most valuable data sources and start collecting your insights, the most fundamental of all data analysis techniques.

4. Set your KPIs

Once you’ve set your data sources, started to gather the raw data you consider to potentially offer value, and established clearcut questions you want your insights to answer, you need to set a host of key performance indicators (KPIs) that will help you track, measure, and shape your progress in a number of key areas.

KPIs are critical to both data analysis methods in qualitative research and data analysis methods in quantitative research. This is one of the primary methods of analyzing data you certainly shouldn’t overlook.

To help you set the best possible KPIs for your initiatives and activities, explore our collection ofkey performance indicator examples.

5. Omit useless data

Having defined your mission and bestowed your data analysis techniques and methods with true purpose, you should explore the raw data you’ve collected from all sources and use your KPIs as a reference for chopping out any information you deem to be useless.

Trimming the informational fat is one of the most crucial steps of data analysis as it will allow you to focus your analytical efforts and squeeze every drop of value from the remaining ‘lean’ information.

Any stats, facts, figures, or metrics that don’t align with your business goals or fit with your KPI management strategies should be eliminated from the equation.

6. Conduct statistical analysis

One of the most pivotal steps of data analysis methods is statistical analysis.

This analysis method focuses on aspects including cluster, cohort, regression, factor, and neural networks and will ultimately give your data analysis methodology a more logical direction.

Here is a quick glossary of these vital statistical analysis terms for your reference:

• Cluster: The action of grouping a set of elements in a way that said elements are more similar (in a particular sense) to each other than to those in other groups, hence the term ‘cluster’.

• Cohort: A subset of behavioral analytics that takes insights from a given data set (e.g. a web application or CMS) and instead of looking at everything as one wider unit, each element is broken down into related groups.

• Regression: A definitive set of statistical processes centered on estimating the relationships among particular variables to gain a deeper understanding of particular trends or patterns.

• Factor: A statistical practice utilized to describe variability among observed, correlated variables in terms of a potentially lower number of unobserved variables called ‘factors’. The aim here is to uncover independent latent variables.

• Neural networks: A neural network is a form of machine learning which is far too comprehensive to summarize, but this explanation will help paint you a fairly comprehensive picture.

7. Build a data management roadmap

While (at this point) this particular step is optional (you will have already gained a wealth of insight and formed a fairly sound strategy by now), creating a data governance roadmap will help your data analysis methods and techniques become successful on a more sustainable basis. These roadmaps, if developed properly, are also built so they can be tweaked and scaled over time.

Invest ample time in developing a roadmap that will help you store, manage, and handle your data internally, and you will make your analysis techniques all the more fluid and functional.

8. Integrate technology

There are many ways to analyze data, but one of the most vital aspects of analytical success in a business context is integrating the right  decision support software and technology.

Robust analysis platforms will not only allow you to pull critical data from your most valuable sources while working with dynamic KPIs that will offer you actionable insights, it will also present the information in a digestible, visual, interactive format from one central, live dashboard. A data analytics methodology you can count on.

By integrating the right technology for your statistical method data analysis and core data analytics methodology, you’ll avoid fragmenting your insights, saving you time and effort while allowing you to enjoy the maximum value from your business’s most valuable insights.

9. Answer your questions

By considering each of the above efforts, working with the right technology, and fostering a cohesive internal culture where everyone buys into the different ways to analyze data as well as the power of digital intelligence, you will swiftly start to answer to your most important, burning business questions. 

10. Visualize your data

Arguably, the best way to make your data analysis concepts accessible across the organization is through data visualization. An online data visualization is a powerful tool as it lets you tell a story with your metrics, allowing users across the business to extract meaningful insights that aid business evolution. It also covers all the different ways to analyze data.

The purpose of data analysis is to make your entire organization more informed and intelligent, and with the right platform or dashboard, this can be simpler than you think.

Data analysis in the big data environment

Big data is invaluable to today’s businesses, and by using different methods for data analysis, it’s possible to view your data in a way that can help you turn insight into positive action.

To inspire your efforts and put the importance of big data into context, here are some insights that could prove helpful. Some facts that will help shape your big data analysis techniques.

• By 2020, around 7 megabytes of new information will be generated every second for every single person on the planet.

• A 10% boost in data accessibility will result in more than $65 million extra net income for your average Fortune 1000 company.

• 90% of the world’s big data was created in the past three years.

• According to Accenture, 79% of notable business executives agree that companies that fail to embrace big data will lose their competitive position and could face extinction. Moreover, 83% of business execs have implemented big data projects to gain a competitive edge.

Data analysis concepts may come in many forms, but fundamentally, any solid data analysis methodology will help to make your business more streamlined, cohesive, insightful and successful than ever before.

Author: Sandra Durcevic

Source: Datapine

Essential Data Science Tools And Frameworks

The fields of data science and artificial intelligence see constant growth. As more companies and industries find value in automation, analytics, and insight discovery, there comes a need for the development of new tools, frameworks, and libraries to meet increased demand. There are some tools that seem to be popular year after year, but some newer tools emerge and quickly become a necessity for any practicing data scientist. As such, here are ten trending data science tools that you should have in your repertoire in 2021.

PyTorch

PyTorch can be used for a variety of functions from building neural networks to decision trees due to the variety of extensible libraries including Scikit-Learn, making it easy to get on board. Importantly, the platform has gained substantial popularity and established community support that can be integral in solving usage problems. A key feature of Pytorch is its use of dynamic computational graphs, which state the order of computations defined by the model structure in a neural network for example.

Scikit-learn

Scikit-learn has been around for quite a while and is widely used by in-house data science teams. Thus it’s not surprising that it’s a platform for not only training and testing NLP models but also NLP and NLU workflows. In addition to working well with many of the libraries already mentioned such as NLTK, and other data science tools, it has its own extensive library of models. Many NLP and NLU projects involve classic workflows of feature extraction, training, testing, model fit, and evaluation, meaning scikit-learn’s pipeline module fits this purpose well. 

CatBoost

Gradient boosting is a powerful machine-learning technique that achieves state-of-the-art results in a variety of practical tasks. For a number of years, it has remained the primary method for learning problems with heterogeneous features, noisy data, and complex dependencies: web search, recommendation systems, weather forecasting, and many others. CatBoost is a popular open-source gradient boosting library with a whole set of advantages, such as being able to incorporate categorical features in your data (like music genre or city) with no additional preprocessing.

Auto-Sklearn

AutoML automatically finds well-performing machine learning pipelines which allow data scientists to focus their efforts on other tasks, reducing the barrier to broadly apply machine learning and makes it available for everyone. Auto-Sklearn frees a machine learning user from algorithm selection and hyperparameter tuning, allowing them to use other data science tools. It leverages recent advantages in Bayesian optimization, meta-learning, and ensemble construction.

Neo4J

As data becomes increasingly interconnected and systems increasingly sophisticated, it’s essential to make use of the rich and evolving relationships within our data. Graphs are uniquely suited to this task because they are, very simply, a mathematical representation of a network. Neo4J is a native graph database platform, built from the ground up to leverage not only data but also data relationships.

Tensorflow

This Google-developed framework excels where many other libraries don’t, such as with its scalable nature designed for production deployment. Tensorflow is often used for solving deep learning problems and for training and evaluating processes up to the model deployment. Apart from machine learning purposes, TensorFlow can be also used for building simulations, based on partial derivative equations. That’s why it is considered to be an all-purpose and one of the more popular data science tools for machine learning engineers.

Airflow

Apache Airflow is a data science tool created by the Apache community to programmatically author, schedule, and monitor workflows. The biggest advantage of Airflow is the fact that it does not limit the scope of pipelines. Airflow can be used for building machine learning models, transferring data, or managing the infrastructure. The most important thing about Airflow is the fact that it is an “orchestrator.” Airflow does not process data on its own, Airflow only tells others what has to be done and when.

Kubernetes

Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications. It groups containers that make up an application into logical units for easy management and discovery. Originally developed by Google, Kubernetes progressively rolls out changes to your application or its configuration, while monitoring application health to ensure it doesn’t kill all your instances at the same time.

Pandas

Pandas is a popular data analysis library built on top of the Python programming language, and getting started with Pandas is an easy task. It assists with common manipulations for data cleaning, joining, sorting, filtering, deduping, and more. First released in 2009, pandas now sits as the epicenter of Python’s vast data science ecosystem and is an essential tool in the modern data analyst’s toolbox.

GPT-3

Generative Pre-trained Transformer 3 (GPT-3) is a language model that uses deep learning to produce human-like text. GPT-3 is the most recent language model coming from the OpenAI research lab team. They announced GPT-3 in a May 2020 research paper, “Language Models are Few-Shot Learners.” While a tool like this may not be something you use daily as an NLP professional, it’s still an interesting skill to have. Being able to spit out human-like text, answer questions, and even create code, it’s a fun factoid to have.

Author: Alex Landa

Source: Open Data Science

“Science has not yet mastered prophecy. We predict too much for the next year and yet far too little for the next ten.”

These words, articulated by Neil Armstrong at a speech to a joint session of Congress in 1969, fit squarely into most every decade since the turn of the century, and it seems to safe to posit that the rate of change in technology has accelerated to an exponential degree in the last two decades, especially in the areas of artificial intelligence and machine learning.

Artificial intelligence is making an extreme entrance into almost every facet of society in predicted and unforeseen ways, causing both excitement and trepidation. This reaction alone is predictable, but can we really predict the associated risks involved?

It seems we’re all trying to get a grip on potential reality, but information overload (yet another side affect that we’re struggling to deal with in our digital world) can ironically make constructing an informed opinion more challenging than ever. In the search for some semblance of truth, it can help to turn to those in the trenches.

In my continued interview with over 30 artificial intelligence researchers, I asked what they considered to be the most likely risk of artificial intelligence in the next 20 years.

Some results from the survey, shown in the graphic below, included 33 responses from different AI/cognitive science researchers. (For the complete collection of interviews, and more information on all of our 40+ respondents, visit the original interactive infographic here on TechEmergence).

Two “greatest” risks bubbled to the top of the response pool (and the majority are not in the autonomous robots’ camp, though a few do fall into this one). According to this particular set of minds, the most pressing short- and long-term risks is the financial and economic harm that may be wrought, as well as mismanagement of AI by human beings.

Dr. Joscha Bach of the MIT Media Lab and Harvard Program for Evolutionary Dynamics summed up the larger picture this way:

“The risks brought about by near-term AI may turn out to be the same risks that are already inherent in our society. Automation through AI will increase productivity, but won’t improve our living conditions if we don’t move away from a labor/wage based economy. It may also speed up pollution and resource exhaustion, if we don’t manage to install meaningful regulations. Even in the long run, making AI safe for humanity may turn out to be the same as making our society safe for humanity.”

Essentially, the introduction of AI may act as a catalyst that exposes and speeds up the imperfections already present in our society. Without a conscious and collaborative plan to move forward, we expose society to a range of risks, from bigger gaps in wealth distribution to negative environmental effects.

Leaps in AI are already being made in the area of workplace automation and machine learning capabilities are quickly extending to our energy and other enterprise applications, including mobile and automotive. The next industrial revolution may be the last one that humans usher in by their own direct doing, with AI as a future collaborator and – dare we say – a potential leader.

Some researchers believe it’s a matter of when and not if. In Dr. Nils Nilsson’s words, a professor emeritus at Stanford University, “Machines will be singing the song, ‘Anything you can do, I can do better; I can do anything better than you’.”

In respect to the drastic changes that lie ahead for the employment market due to increasingly autonomous systems, Dr. Helgi Helgason says, “it’s more of a certainty than a risk and we should already be factoring this into education policies.”

Talks at the World Economic Forum Annual Meeting in Switzerland this past January, where the topic of the economic disruption brought about by AI was clearly a main course, indicate that global leaders are starting to plan how to integrate these technologies and adapt our world economies accordingly – but this is a tall order with many cooks in the kitchen.

Another commonly expressed risk over the next two decades is the general mismanagement of AI. It’s no secret that those in the business of AI have concerns, as evidenced by the $1 billion investment made by some of Silicon Valley’s top tech gurus to support OpenAI, a non-profit research group with a focus on exploring the positive human impact of AI technologies.

“It’s hard to fathom how much human-level AI could benefit society, and it’s equally hard to imagine how much it could damage society if built or used incorrectly,” is the parallel message posted on OpenAI’s launch page from December 2015. How we approach the development and management of AI has far-reaching consequences, and shapes future society’s moral and ethical paradigm.

Philippe Pasquier, an associate professor at Simon Fraser University, said “As we deploy more and give more responsibilities to artificial agents, risks of malfunction that have negative consequences are increasing,” though he likewise states that he does not believe AI poses a high risk to society on its own.

With great responsibility comes great power, and how we monitor this power is of major concern.

Dr. Pei Wang of Temple University sees major risk in “neglecting the limitations and restrictions of hot techniques like deep learning and reinforcement learning. It can happen in many domains.” Dr. Peter Voss, founder of SmartAction, expressed similar sentiments, stating that he most fears “ignorant humans subverting the power and intelligence of AI.”

Thinking about the risks associated with emerging AI technology is hard work, engineering potential solutions and safeguards is harder work, and collaborating globally on implementation and monitoring of initiatives is the hardest work of all. But considering all that’s at stake, I would place all my bets on the table and argue that the effort is worth the risk many times over.

Source: Tech Crunch

Facing the major challenges that come with big data

Worldwide, over 2.5 quintillion bytes of data are created every day. And with the expansion of the Internet of Things (IoT), that pace is increasing. 90% of the current data in the world was generated in the last two years alone. When it comes to businesses, for a forward thinking, digitally transforming organization, you’re going to be dealing with data. A lot of data. Big data.

Challenges faced by businesses

While simply collecting lots of data presents comparatively few problems, most businesses run into two significant roadblocks in its use: extracting value and ensuring responsible handling of data to the standard required by data privacy legislation like GDPR. What most people don’t appreciate is the sheer size and complexity of the data sets that organizations have to store and the related IT effort, requiring teams of people working on processes to ensure that others can access the right data in the right way, when they need it, to drive essential business functions. All while ensuring personal information is treated appropriately.

The problem comes when you’ve got multiple teams around the world, all running to different beats, without synchronizing. It’s a bit like different teams of home builders, starting work independently, from different corners of a new house. If they have all got their own methods and bricks, then by the time they meet in the middle, their efforts won’t match up. It’s the same in the world of IT. If one team is successful, then all teams should be able to learn those lessons of best practice. Meanwhile, siloed behavior can become “free form development” where developers write code to suit a specific problem that their department is facing, without reference to similar or diverse problems that other departments may be experiencing.

In addition, often there simply aren’t enough builders going around to get these data projects turned around quickly, which can be a problem in the face of heightening business demand. In the scramble to get things done at the pace of modern business, at the very least there will be some duplication of effort, but there’s also a high chance of confusion and the foundations for future data storage and analysis won’t be firm. Creating a unified, standard approach to data processing is critical – as is finding a way to implement it with the lowest possible level of resource, at the fastest possible speeds.

Data Vault automation

One of the ways businesses can organize data to meet both the needs for standardization and flexibility is in a Data Vault environment. This data warehousing methodology is designed to bring together information from multiple different teams and systems into a centralized repository, providing a bedrock of information that teams can use to make decisions – it includes all of the data, all of the time, ensuring that no information is missed out of the process.

However, while a Data Vault design is a good architect’s drawing, it won’t get the whole house built on its own. Developers can still code and build it manually over time but given its complexity they certainly cannot do this quickly, and potentially may not be able to do it in a way that can stand up to the scrutiny of data protection regulations like GDPR. Building a Data Vault environment by hand, even using standard templates, can be incredibly laborious and potentially error prone.

This is where Data Vault automation comes in, taking care of the 90% or so of an organization’s data infrastructure that fits standardized templates and the stringent requirements that the Data Vault 2.0 methodology demands. Data vault automation can lay out the core landscape of a Data Vault, as well as make use of reliable, consistent metadata to ensure information, including personal information, can be monitored both at its source and over time as records are changed.

Author: Dan Linstedt

Source: Insidebigdata

Finding your way in programming: top 10 languages summarize

The landscape of programming languages is rich and expanding, which can make it tricky to focus on just one or another for your career. We highlight some of the most popular languages that are modern, widely used, and come with loads of packages or libraries that will help you be more productive and efficient in your work.

1. Python — Artificial Intelligence & Machine Learning

• Level: Beginner
• Popular Frameworks: Django, Flask
• Platform: Web, Desktop
• Popularity: #1 on PYPL Popularity Index of March 2021, #3 on Tiobe Index for March 2021, Loved by 66.7% of StackExchange developers in 2020, and wanted by 30%, the most of any language.

Developed by Guido van Rossum in the 1990s, the multi-purpose high-level Python has grown extremely fast over the years to become one of the most popular programming languages today.

And the number one reason for Python’s popularity is its beginner-friendliness, which allows anyone, even individuals with no programming background, to pick up Python and start creating simple programs.

But that’s not all. It also offers an exceptionally vast collection of packages and libraries that can play a key role in reducing the ETA for your projects, along with a strong community of like-minded developers that is eager to help.

What this language is used for — 

Although Python can be used to build pretty much anything, it really shines when it comes to working on technologies like Artificial Intelligence, Machine Learning, Data Analytics. Python also proves to be useful for web development, creating enterprise applications, and GUIs for applications.

Python is used in many application domains. Here’s a sampling — 

https://www.python.org/about/apps/

 Additional Resources:

• Learn Python — freecodecamp
• Python Tutorial — Python for Beginners — Programming With Mosh
• Python Tutorial — Learnpython.org

2. JavaScript — Rich Interactive Web Development

• Level: Beginner
• Popular Frameworks:React.js, Vue, Meteor
• Platform: Web, Desktop, Frontend scripting
• Popularity: #3 on PYPL Popularity Index of March 2021, #7 on Tiobe Index for March 2021, Loved by 58.3% of StackExchange developers in 2020, and wanted by 18.5%, the most of any language.

JavaScript was one of the key programming languages alongside HTML and CSS that helped build the internet. JavaScript was created in 1995 by Netscape, the company that released the famous Netscape Navigator browser, to eliminate the crudeness of static web pages and add a pinch of dynamic behavior to them.

Today, JavaScript has become a high-level multi-paradigm programming language that serves as the world’s top frontend programming language for the web, handling all the interactions offered by the webpages, such as pop-ups, alerts, events, and many more like them.

What this language is used for — 

JavaScript is the perfect option if you want your app to run across a range of devices, such as smartphones, cloud, containers, micro-controllers, and on hundreds of browsers. For the server-side workloads, there’s Node.js, a proven JavaScript runtime that is being used by thousands of companies today.

Additional Resources:

• Learn JavaScript — freecodecamp
• JavaScript Tutorial for Beginners: Learn JavaScript in 1 Hour — Programming with Mosh
• Learn JavaScript By Building Seven Games — freecodecamp

3. Java — Enterprise Application Development

• Level: Intermediate
• Popular Frameworks: Spring, Hibernate, Strut
• Platform: Web, Mobile, Desktop
• Popularity: #2 on PYPL Popularity Index of March 2021, #2 on Tiobe Index for March 2021, Loved by 44.1% of StackExchange developers in 2020.

Java has remained the de-facto programming language for building enterprise-grade applications for more than 20 years now.

Created by Sun Microsystems’ James Gosling in 1995, the object-oriented programming language Java has been serving as a secure, reliable, and scalable tool for developers ever since.

Some of the features offered by Java that make it more preferable than several other programming languages are its garbage collection capabilities, backward compatibility, platform independence via JVM, portability, and high performance.

Java’s popularity can be seen clearly among the Fortune 500 members as 90% of them use Java to manage their business efficiently.

What this language is used for — 

Apart from being used to develop robust business applications, Java has also been used extensively in Android, making it a prerequisite for Android developers. Java also allows developers to create apps for a range of industries, such as banking, electronic trading, e-commerce, as well as apps for distributed computing.

Additional Resources:

• Learn Java — Codecademy
• Learn Java Programming — Programiz

4. R — Data Analysis

• Level: Intermediate
• Popular Studio: R Studio
• Platform: Mainly desktop
• Popularity: #7 on PYPL Popularity Index of March 2021.

If you do any sort of data analysis or work on Machine Learning projects, the chances are that you may have heard about R. The R programming language was first released to the public in 1993 by its creators Ross Ihaka and Robert Gentleman as an implementation of the S programming language with a special focus on statistical computing and graphical modeling.

Over the years, R became one of the best programming languages for projects requiring extensive data analysis, graphical data modeling, spatial and time-series analysis.

R also provides great extensibility via its functions and extensions that offer a ton of specialized techniques and capabilities to developers. The language also works remarkably well with code from other programming languages, such as C, C++, Python, Java, and .NET.

What this language is used for — 

Apart from some of the uses mentioned above, R can be used for behavior analysis, data science, and machine learning projects that involve classification, clustering, and more.

Additional Resources:

• R Programming Tutorial — Learn the Basics of Statistical Computing — freecodecamp
• R Programming — Coursera
• Learn R — Codecademy

5. C/C++ — Operating Systems and System Tools

• Level: C — Intermediate to Advanced, C++ — Beginner to Intermediate
• Popular Frameworks: MFC, .Net, Qt, KDE, GNOME
• Platform: Mobile, Desktop, Embedded

Believe it or not, the programming languages C/C++ were all the rage in the very late 20th century. Why?

It’s because C and C++ are both very low-level programming languages, offering blazing fast performance, which is why they were and are still being used to develop operating systems, file systems, and other system-level applications.

While C was released in the 70s by Dennis Ritchie, C++, an extension to C with classes and many other additions, such as object-oriented features, was released later by Bjarne Stroustrup in the mid-80s. 

Even after close to 50 years, both the programming languages are still being used to create rock-steady and some of the fastest applications of all times.

What this language is used for — 

As C & C++ both offer full access to the underlying hardware, they have been used to create a wide variety of applications and platforms, such as system applications, real-time systems, IoT, embedded systems, games, cloud, containers, and more.

Additional Resources: 

• C Programming Tutorial for Beginners — freecodecamp
• C++ Tutorial for Beginners — Full Course — freecodecamp
• Learn C++ — Codecademy
• Learn C — Programiz

6. Golang — Server-Side Programming

• Level: Beginner to intermediate
• Popular Framework: Revel, Beego
• Platform: Cross-platform, mainly desktop
• Popularity: Loved by 62.3% of StackExchange developers in 2020, and wanted by 17.9%, the most of any language.

Go, or Golang, is a compiled programming language developed by the search giant Google. Created in 2009, Golang is an effort by the designers at Google to eliminate all the faults in the languages used throughout the organization and by keeping all the best features intact.

Golang is fast and has a simple syntax, allowing anyone to pick up the programming language. It also comes with cross-platform support, making it easy and efficient to use.

Go claims to offer a mix of high-performance like C/C++, simplicity, and usability like Python, along with efficient concurrency handling like Java.

What this language is used for — 

Go is primarily used in back-end technologies, cloud services, distributed networks, IoT, but it has also been used to create console utilities, GUI applications, and web applications.

Additional Resources :

• Golang Tutorial for Beginners— freecodecamp
• Go Tutorial — Tutorialspoint
• Introducing Go — Caleb Doxsey

7. C# — Application & Web Development Using .NET

• Level: Intermediate
• Popular Frameworks: .NET, Xamarin
• Platform: Cross-platform, including mobile and enterprise software applications
• Popularity: #4 on PYPL Popularity Index of March 2021, #5 on Tiobe Index for March 2021, Loved by 59.7% of StackExchange developers in 2020.

C# was Microsoft’s approach to developing a programming language similar to the object-oriented C as part of its .NET initiative. The general-purpose multi-paradigm programming language was unveiled in 2000 by Anders Hejlsberg and has a syntax similar to C, C++, and Java. 

This was a huge plus point for developers who were familiar with either of these languages. It also offered relatively faster compilation and execution along with seamless scalability.

C# was designed keeping in mind the .NET ecosystem, which allows developers to access a range of libraries and frameworks offered by Microsoft. And with the integration with Windows, C# becomes extremely easy to use, even perfect for developing Windows-based apps.

What this language is used for — 

Developers can use C# for a range of projects, including game development, server-side programming, web development, creating web forms, mobile applications, and more. C# has also been used to develop apps for the Windows platform, specifically Windows 8 and 10.

Additional Resources: 

• Learn C# — Codeacademy
• C# Tutorials — W3Schools

8. PHP — Web Development

• Level: Beginner
• Popular Frameworks: CakePHP, Larawell, Symfony, Phalcon
• Platform: Cross-platform (desktop, mobile, web) Back-end web scripting.
• Popularity: #6 on PYPL Popularity Index of March 2021, #8 on Tiobe Index for March 2021.

Just like Guido van Rossum’s Python, PHP also came to fruition as a side project by Rasmus Lerdorf, with the initial development dating back to the year 1994.

Rasmus’s version of PHP was originally intended to help him maintain his personal homepage, but over the years, the project evolved to support web forms and databases.

Today, PHP has become a general-purpose scripting language that’s being used around the globe, primarily for server-side web development. It is fast, simple, and is platform-independent, along with a large open-source software community.

What this language is used for — 

A large number of companies are using PHP today to create tools like CMS (Content Management Systems), eCommerce platforms, and web applications. PHP also makes it extremely easy to create web pages in an instant.

9. SQL — Data Management

• Level: Beginner
• Platform: Back-end database management
• Popularity: #10 on Tiobe Index for March 2021, Loved by 56.6% of StackExchange developers in 2020.

SQL, short for Structured Query Language, is probably one of the most crucial programming languages on this list.

Designed by Donald D. Chamberlin and Raymond F. Boyce in 1974, the special-purpose programming language has played a key role in enabling developers to create and manage tables and databases for storing relational data over hundreds of thousands of data fields.

Without SQL, organizations would have to rely on older and possibly slower methods of storing and accessing vast amounts of data. With SQL, much of these tasks can be done within seconds.

Over the years, SQL has helped spawn a large number of RDBMS (Relational Database Management Systems) that offer much more than just the creation of tables and databases.

What this language is used for — 

Pretty much every other project or industry that needs to deal with large amounts of data stored in tables or databases uses SQL through an RDBMS.

Additional Resources:

• Learn SQL — Codecademy
• NoSQL Databases Explained — IBM Cloud
• Coding Resources: SQL — Berkeley Boot Camps

10. Swift — For Mobile App Development on iOS

• Level: Beginner
• Popular Frameworks: Alamofire, RxSwift, Snapkit
• Platform: Mobile (Apple iOS apps, specifically)
• Popularity: #9 on PYPL Popularity Index of March 2021, Loved by 59.5% of StackExchange developers in 2020.

Apple’s full control over its hardware and software has allowed it to deliver smooth and consistent experiences across its range of devices. And that’s where Swift comes in.

Swift is Apple’s own programming language that was released in 2014 as a replacement for its Objective-C programming language. It is a multi-paradigm general-purpose programming language that’s extremely efficient and designed to improve developer productivity.

Swift is a modern programming language (newest on this list), fast, powerful, and offers full interoperability with Objective-C. Over the years, Swift received numerous updates that helped it gain significant popularity among Apple’s iOS, macOS, watchOS, and tvOS platforms.

What this language is used for — 

Paired with Apple’s Cocoa and Cocoa Touch framework, Swift can be used to create apps for virtually every Apple device, such as iPhones, iPads, Mac, Watch, and other devices.

Additional Resources — 

• Swift Programming Tutorial for Beginners (Full Tutorial) — CodeWithChris

Conclusion

Now let’s quickly conclude this article by giving you an insight into the importance and career growth opportunities associated with these programming languages. Every programming language has its own set of benefits, and out of all the entries, you can enter the field of your choice.

Mastering Python can help you land one of the top 3 highest paying job roles in the industry. With Python, you can apply for Software Engineer, DevOps Engineer, Data Scientist, and can even secure job positions in the most reputed companies with a handsome package.

You can simply opt for Quantitative Analyst, Data Visualization, Expert, Business Intelligence Expert, and Data Analyst with R.

Regarding JavaScript, there is a high demand for Javascript developers offering a modest salary.

But there’s no beating the efficiency of C/C++ when it comes to building system tools and operating systems as it continues to enjoy the number one spot on TIOBE’s software quality index. SQL remains one of the best programming languages to tinker around vast databases, while C# proves perfect for Windows. Swift has also been seeing a rise in popularity among developers looking to build for Apple’s hardware. As for PHP and Go, they continue to maintain a respectable position in the industry.

So, out of the 10 programming languages, it’s totally up to you which one is your go-to choice and makes your career in. So choose wisely!

Author: Claire D. Costa

Source: KDnuggets

Launching an effective digital analytics strategy is a must-do to understand your customers. But many organizations are still trying to figure out how to get business values from expensive analytics programs. Here are 5 common analytics mistakes that can kill any predictive analytics effort.

Why predictive analytics projects fail

Predictive Analytics is becoming the next big buzzword in the industry. But according to Mike Le, co-founder and chief operating officer at CB/I Digital in New York, implementing an effective digital analytics strategy has proven to be very challenging for many organizations. “First, the knowledge and expertise required to setup and analyze digital analytics programs is complicated,” Le notes. “Second, the investment for the tools and such required expertise could be high. Third, many clients see unclear returns from such analytics programs. Learning to avoid common analytics mistakes will help you save a lot of resources to focus on core metrics and factors that can drive your business ahead.” Here are 5 common mistakes that Le says cause many predictive analytics projects to fail.

Mistake 1: Starting digital analytics without a goal

“The first challenge of digital analytics is knowing what metrics to track, and what value to get out of them,” Le says. “As a result, we see too many web businesses that don’t have basic conversion tracking setup, or can’t link the business results with the factors that drive those results. This problem happens because these companies don’t set a specific goal for their analytics. When you do not know what to ask, you cannot know what you'll get. The purpose of analytics is to understand and to optimize. Every analytics program should answer specific business questions and concerns. If your goal is to maximize online sales, naturally you’ll want to track the order volume, cost-per-order, conversion rate and average order value. If you want to optimize your digital product, you’ll want to track how users are interact with your product, the usage frequency and the churn rate of people leaving the site. When you know your goal, the path becomes clear.”

Mistake 2: Ignoring core metrics to chase noise

“When you have advanced analytics tools and strong computational power, it’s tempting to capture every data point possible to ‘get a better understanding’ and ‘make the most of the tool,’” Le explains. “However, following too many metrics may dilute your focus on the core metrics that reveal the pressing needs of the business. I've seen digital campaigns that fail to convert new users, but the managers still setup advanced tracking programs to understand user 

behaviors in order to serve them better. When you cannot acquire new users, your targeting could be wrong, your messaging could be wrong or there is even no market for your product - those problems are much bigger to solve than trying to understand your user engagement. Therefore, it would be a waste of time and resources to chase fancy data and insights while the fundamental metrics are overlooked. Make sure you always stay focus on the most important business metrics before looking broader.”

Mistake 3: Choosing overkill analytics tools

“When selecting analytics tools, many clients tend to believe that more advanced and expensive tools can give deeper insights and solve their problems better,” Le says. “Advanced analytics tools may offer more sophisticated analytic capabilities over some fundamental tracking tools. But whether your business needs all those capabilities is a different story. That's why the decision to select an analytics tool should be based on your analytics goals and business needs, not by how advanced the tools are. There’s no need to invest a lot of money on big analytics tools and a team of experts for an analytics program while some advanced features of free tools like Google Analytics can already give you the answers you need.”

Mistake 4: Creating beautiful reports with little business value

“Many times you see reports that simply present a bunch of numbers exported from tools, or state some ‘insights’ that has little relevance to the business goal,” Le notes. “This problem is so common in the analytics world, because a lot of people create reports for the sake of reporting. They don’t think about why those reports should exist, what questions they answer and how those reports can add value to the business. Any report must be created to answer a business concern. Any metrics that do not help answer business questions should be left out. Making sense of data is hard. Asking right questions early will

help.”

Mistake 5: Failing to detect tracking errors

“Tracking errors can be devastating to businesses, because they produce unreliable data and misleading analysis,” Le cautions. “But many companies do not have the skills to setup tracking properly, and worse, to detect tracking issues when they happen. There are many things that can go wrong, such as a developer mistakenly removing the tracking pixels, transferring incorrect values, the tracking code firing unstably or multiple times, wrong tracking rule's logic, etc. The difference could be so subtle that the reports look normal, or are only wrong in certain scenarios. Tracking errors easily go undetected because it takes a mix of marketing and tech skills. Marketing teams usually don’t understand how tracking works, and development teams often don’t know what ‘correct’ means. To tackle this problem, you should frequently check your data accuracy and look for unusual signs in reports. Analysts should take an extra step to learn the technical aspect of tracking, so they can better sense the problems and raise smart questions for the technical team when the data looks suspicious.”

Author: Mike Le

Source: Information Management

Four important drivers of data science developments

According to the Gartner Group, digital business reached a tipping point last year, with 49% of CIOs reporting that their enterprises have already changed their business models or are in the process of doing so. When Gartner asked CIOs and IT leaders which technologies they expect to be most disruptive, artificial intelligence (AI) was the top-mentioned technology.

AI and ML are having a profound impact on enterprise digital transformation becoming crucial as a competitive advantage and even for survival. As the field grows, four trends emerge, shaping data science in the next five years:

Accelerate the full data science life-cycle

The pressure to grow ROI from AI and ML initiatives has pushed demand for new innovative solutions that accelerate AI and data science. Although data science processes are iterative and highly manual, more than 40% of data science tasks are expected to be automated by 2020, according to Gartner, resulting in increased productivity and broader usage of data across the enterprise.

Recently, automated machine learning (AutoML) has become one of the fastest-growing technologies for data science. Machine learning, however,  typically accounts for only 10-20% of the entire data science process. Real pains exist before the machine learning stage with data and feature engineering.  The new concept of data science automation goes beyond machine learning automation, including data preparation, feature engineering, machine learning, and the production of full data science pipelines. With data science automation, enterprises can genuinely accelerate AI and ML initiatives.

Leverage existing resources for democratization

Despite substantial investments in data science across many industries, the scarcity of data science skills and resources often limits the advancement of AI and ML projects in organizations. The shortage of data scientists has created a challenge for anyone implementing AI and ML initiatives, forcing a closer look at how to build and leverage data science resources.

Other than the need for highly specialized technical skills and mathematical aptitude, data scientists must also couple these skills with domain/industry knowledge that is relevant to a specific business area. Domain knowledge is required for problem definition and result validation and is a crucial enabler to deliver business value from data science. Relying on 'data science unicorns' that have all these skill sets is neither realistic nor scalable.

Enterprises are focusing on repurposing existing resources as 'citizen' data scientists. The rise of AutoML and data science automation can unlock data science to a broader user base and allow the practice to scale. By empowering citizen data scientists allowing them to execute standard use cases, skilled data scientists can focus on high-impact, technically-challenging projects to produce higher values.

Augment insights for greater transparency

As more organizations are adopting data science in their business process, relying on AI-derived recommendations that lack transparency is becoming problematic. Increased regulatory oversight like the GDPR has exacerbated the problem. Transparent insights make AI models more 'oversight' friendly and have the added benefit of being far more actionable.

White-box AI models help organizations maintain accountability in data-driven decisions and allow them to live within the boundaries of regulations. The challenge is the need for high-quality and transparent inputs (aka 'features'), often requiring multiple manual iterations to achieve the needed transparency. Data science automation allows data scientists to explore millions of hypotheses and augments their ability to discover transparent and predictive features as business insights.

Operationalize data science in business

Although ML models are often tiny pieces of code, when models are finally deemed ready for production, deploying them can be complicated and problematic. For example, since data scientists are not software engineers, the quality of their code may not be production-ready. Data scientists often validate the models with down-sampled datasets in labs environments and models may not be scalable enough for production-scale datasets. Also, the performance of deployed models decreases as data invariably changes, making model maintenance pivotal to extract business value from AI and ML models continuously. Data and feature pipelines are much bigger and more complex than ML models themselves, and operationalizing data and feature pipelines is even more complicated.  One of the promising approaches is to leverage concepts from continuous deployment through APIs. Data science automation can generate APIs to execute the full data science pipeline, accelerating deployments while also providing an ongoing connection to development systems to accelerate the optimization and maintenance of models.

Data science is at the heart of AI and ML. While the promise of AI is real, the problems associated with data science are also real. Through better planning, closer cooperation with line of business and by automating the more tedious and repetitive parts of the process, data scientists can finally begin to focus on what to solve, rather than how to solve.

Author: Daniel Gutierrez

Source: Insidebigdata

Functions and applications of generative AI models

Learn how industries use generative AI models, which function on their own to create new content and alongside discriminative models to identify, for example, 'real' vs. 'fake.'

AI encompasses many techniques for developing software models that can accomplish meaningful work, including neural networks, genetic algorithms and reinforcement learning. Previously, only humans could perform this work. Now, these techniques can build different kinds of AI models.

Generative AI models are one of the most important kinds of AI models. A generative model creates things. Any tool that uses AI to generate a new output -- a new picture, a new paragraph or a new machine part design -- incorporates a generative model.

The various applications for generative models

Generative AI functions across a broad spectrum of applications, including the following:

• Natural language interfaces. In performing both speech and text synthesis, these AI systems power digital assistants such as Amazon's Alexa, Apple's Siri and Google Assistant, as well as tools that auto-summarize text or autogenerate press releases from a set of key facts.
• Image synthesis. These AI systems create images based on instructions or directions. They will, if told to, create an image of a kiwi bird eating a kiwi fruit while sitting on a big padlock key. They can be used to create ads, fashion designs or movie production storyboards. DALL-E, Midjourney and Wombo Dream are examples of AI image generators.
• Space synthesis. AI can also create three-dimensional spaces and objects, both real and digital. It can design buildings, rooms and even whole city plans, as well as virtual spaces for gameplay or metaverse-style collaboration. Spacemaker is a real-world architectural program, while Meta's BuilderBot (in development) will focus on virtual spaces.
• Product design and object synthesis. Now that the public is more aware of 3D printing, it's worth noting that generative AI can design and even create physical objects like machine parts and household goods. AutoCAD and SOL75 are tools using AI to perform or assist in physical object design.

Many tools harness both generative and discriminative AI models. Discriminative models, adversely, identify things. Any tool that uses AI to identify, categorize, tag or assess the authenticity of an artifact (physical or digital) incorporates a discriminative model. A discriminative model typically doesn't say categorically what something is, but rather what it most likely is based on what it sees.

How generative and discriminative models function together

A generative adversarial network (GAN) uses a generative model to create outputs and an adversarial discriminative model to evaluate them, with feedback loops between the two. For example, a GAN might be tasked with writing fake restaurant reviews. The generative model would attempt to create seemingly real reviews, then pass them, along with real reviews, through the discriminative model. The discriminator acts as an adversary to the generative model, trying to identify the fakes.

The feedback loops ensure that the exercise trains both models to perform better. The discriminator, which is then told which inputs were real and which were fake after evaluating them, adjusts itself to get better at identifying fakes and not flagging real reviews as fake. The generator gets better at generating undetectable fakes as it learns which fakes the discriminator successfully identified and which authentic reviews it incorrectly tagged.

This phenomenon is applied in the following industries:

• Finance. AI systems watch transaction streams in real time and analyze them in the context of a person's history to judge whether a transaction is authentic or fraudulent. All major banks and credit card companies use such software now; some develop their own and others use commercially available solutions.
• Manufacturing. Factory AI systems can watch streams of inputs and outputs using cameras, x-rays, etc. They can flag or deflect parts and products likely to be defective. Kyocera Communications and Foxconn both use AI for visual inspection in their facilities.
• Film and media. Just as generative tools can create fake images (e.g., a kiwi bird eating kiwi on a key), discriminative AI can identify faked images or audio files. Google's Jigsaw division focuses in part on developing technology to make deepfake detection more reliable and easier.
• Social media and tech industry. AI systems can look at postings and patterns in postings to help spot fake accounts by disinformation bots or other bad actors. Meta has used AI for years to help find fake accounts and to flag or block COVID misinformation related to the pandemic.

Generative AI may well become a widely known tech buzzword, like automation, and its myriad applications prove that this nascent branch of AI is here to stay. To meet modern challenges facing the tech industry, it only makes sense that this technology will expand and become deeply embedded in more and more enterprises.

Author: John Burke

Source: TechTarget

Gaining advantages with the IoT through 'Thing Management'

Some are calling the industrial Internet of Things the next industrial revolution, bringing dramatic changes and improvements to almost every sector. But to be sure it’s successful, there is one big question: how can organizations manage all the new things that are part of their organizations’ landscapes?

Most organizations see asset management as the practice of tracking and managing IT devices such as routers, switches, laptops and smartphones. But that’s only part of the equation nowadays. With the advent of the IoT, enterprise things now include robotic bricklayers, agitators, compressors, drug infusion pumps, track loaders, scissor lifts and the list goes on and on, while all these things are becoming smarter and more connected.

These are some examples for specific industries:

● Transportation is an asset-intensive industry that relies on efficient operations to achieve maximum profitability. To help customers manage these important assets, GE Transportation is equipping its locomotives with devices that manage hundreds of data elements per second. The devices decipher locomotive data and uncover use patterns that keep trains on track and running smoothly.

● The IoT’s promise for manufacturing is substantial. The IoT can build bridges that help solve the frustrating disconnects among suppliers, employees, customers, and others. In doing so, the IoT can create a cohesive environment where every participant is invested in and contributing to product quality and every customer’s feedback is learned from. Smart sensors, for instance, can ensure that every item, from articles of clothing to top-secret defense weapons, can have the same quality as the one before. The only problem with this is that the many pieces of the manufacturing puzzle and devices in the IoT are moving so quickly that spreadsheets and human analysis alone are not enough to manage the devices.

● IoT in healthcare will help connect a multitude of people, things with smart sensors (such as wearables and medical devices), and environments. Sensors in IoT devices and connected “smart” assets can capture patient vitals and other data in real time. Then data analytics technologies, including machine learning and artificial intelligence (AI), can be used to realize the promise of value-based care. There’s significant value to be gained, including operational efficiencies that boost the quality of care while reducing costs, clinical improvements that enable more accurate diagnoses, and more.

● In the oil and gas industry, IoT sensors have transformed efficiencies around the complex process of natural resource extraction by monitoring the health and efficiency of hard-to-access equipment installations in remote areas with limited connectivity.

● Fuelled by greater access to cheap hardware, the IoT is being used with notable success in logistics and fleet management by enabling cost-effective GPS tracking and automated loading/unloading.

All of these industries will benefit from the IoT. However, as the IoT world expands, these industries and others are looking for ways to track the barrage of new things that are now pivotal to their success. Thing Management pioneers such as Oomnitza help organizations manage devices as diverse as phones, fork lifts, drug infusion pumps, drones and VR headset, providing an essential service as the industrial IoT flourishes.

Think IoT, not IoP

To successfully manage these Things, enterprises are not only looking for Thing Management. They also are rethinking the Internet, not as the Internet of People (IoP), but as the Internet of Things (IoP). Things aren’t people, and there are three fundamental differences.

Many more things are connected to the Internet than people

John Chambers, former CEO of Cisco, recently declared there will be 500 billion things connected by 2024. That’s nearly 100 times the number of people on the planet.

Things have more to say than people

A typical cell phone has nearly 14 sensors, including an accelerometer, GPS, and even a radiation detector. Industrial things such as wind turbines, gene sequencers, and high-speed inserters can easily have over 100 sensors.

Things can speak much more frequently

People enter data at a snail’s pace when compared to the barrage of data coming from the IoT. A utility grid power sensor, for instance, can send data 60 times per second, a construction forklift once per minute, and a high-speed inserter once every two seconds.

Technologists and business people both need to learn how to collect and put all of the data coming from the industrial IoT to use and manage every connected thing. They will have to learn how to build enterprise software for things versus people.

How the industrial IoT will shape the future

The industrial IoT is all about value creation: increased profitability, revenue, efficiency, and reliability. It starts with the target of safe, stable operations and meeting environmental regulations, translating to greater financial results and profitability.

But there’s more to the big picture of the IoT than that. Building the next generation of software for things is a worthy goal, with potential results such as continually improving enterprise efficiency and public safety, driving down costs, decreasing environmental impacts, boosting educational outcomes and more. Companies like GE, Oomnitza and Bosch are investing significant amounts of money in the ability to connect, collect data, and learn from their machines.

The IoT and the next generation of enterprise software will have big economic impacts as well. The cost savings and productivity gains generated through “smart” thing monitoring and adaptation are projected to create $1.1 trillion to $2.5 trillion in value in the health care sector, $2.3 trillion $11.6 trillion in global manufacturing, and $500 billion $757 billion in municipal energy and service provision over the next decade. The total global impact of IoT technologies could generate anywhere from $2.7 trillion to $14.4 trillion in value by 2025.

Author: Timothy Chou

Source: Information-management

Gaining control of big data with the help of NVMe

Every day there is an unfathomable amount of data, nearly 2.5 quintillion bytes, being generated all around us. Part of the data being created we see every day, such as pictures and videos on our phones, social media posts, banking and other apps.

In addition to this, there is data being generated behind the scenes by ubiquitous sensors and algorithms, whether that’s to process quicker transactions, gain real-time insights, crunch big data sets or to simply meet customer expectations. Traditional storage architectures are struggling to keep up with all this data creation, leading IT teams to investigate new solutions to keep ahead and take advantage of the data boom.

Some of the main challenges are understanding performance, removing data throughput bottlenecks and being able to plan for future capacity. Architecture can often lock businesses in to legacy solutions, and performance needs can vary and change as data sets grow.

Architectures designed and built around NVMe(non-volatile memory express) can provide the perfect balance, particularly for data-intensive applications that demand fast performance. This is extremely important for organizations that are dependent on speed, accuracy and real-time data insights.

Industries such as healthcare, autonomous vehicles, artificial intelligence(AI)/machine learning(ML) and Genomics are at the forefront of the transition to high performance NVMe storage solutions that deliver fast data access for high performance computing systems that drive new research and innovations.

Genomics

With traditional storage architectures, detailed genome analysis can take upwards of five days to complete, which makes sense considering an initial analysis of one person’s genome produces approximately 300GB - 1TB of data, and a single round of secondary analysis on just one person’s genome can require upwards of 500TB storage capacity. However, with an NVMe solution implemented it’s possible to get results in just one day.

In a typical study, genome research and life sciences companies need to process, compare and analyze the genomes of between 1,000 and 5,000 people per study. This is a huge amount of data to store, but it’s imperative that it’s done. These studies are working toward revolutionary scientific and medical advances, looking to personalize medicine and provide advanced cancer treatments. This is only now becoming possible thanks to the speed that NVMe enables researchers to explore and analyze the human genome.

Autonomous vehicles

A growing trend in the tech industry is the one of autonomous vehicles. Self-driving cars are the next big thing, and various companies are working tirelessly to perfect the idea. In order to function properly, these vehicles need very fast storage to accelerate the applications and data that ‘drive’ autonomous vehicle development. Core requirements for autonomous vehicle storage include:

• Must have a high capacity in a small form factor
• Must be able to accept input data from cameras and sensors at “line rate” – AKA have extremely high throughput and low latency
• Must be robust and survive media or hardware failures
• Must be “green” and have minimal power footprint
• Must be easily removable and reusable
• Must use simple but robust networking

What kind of storage meets all these requirements? That’s right – NVMe.

Artificial Intelligence

Artificial Intelligence (AI) is gaining a lot of traction in a variety of industries varying from financial to manufacturing, and beyond. In financial, AI does things like predict investment trends. In manufacturing, AI-based image recognition software checks for defects during product assembly. Wherever it’s used, AI needs a high level of computing power, coupled with a high-performance and low-latency architecture in order to enable parallel processing power of data in real-time.

Once again, NVMe steps up to the plate, providing the speed and processing power that is critical during training and inference. Without NVMe to prevent bottlenecks and latency issues, these stages can take much, much longer. Which, in turn, can lead to the temptation to take shortcuts, causing software to malfunction or make incorrect decisions down the line.

The rapid increase of data creation has put traditional storage architectures under high pressure due to its lack of scalability and flexibility, both of which are required to fulfill future capacity and performance requirements. This is where NVMe comes in, breaking the barriers of existing designs by offerings unanticipated density and performance. The breakthroughs that NVMe is able to offer contain the requirements needed to help manage and maintain the data boom.

Author: Ron Herrmann

Source: Dataversity

Research firm Gartner has identified five "cool vendors" in the data science and machine learning space, identifying the features that make their products especially unique or useful. The report, "5 Cool Vendors in Data Science and Machine Learning" was written by analysts Peter Krensky, Svetlana Sicular, Jim Hare, Erick Brethenoux and Austin Kronz. Here are the highlights of what they had to say about each vendor.

DimensionalMechanics

Bellevue, Washington
www.dimensionalmechanics.com
“DimensionalMechanics has built a data science platform that breaks from market traditions; where more conventional vendors have developed work flow-based or notebook-based data science environments, DimensionalMechanics has opted for a “data-science metalanguage,” Erick Brethenoux writes. “In effect, given the existing use cases the company has handled so far, its NeoPulse Framework 2.0 acts as an “AutoDL” (Auto-Deep Learning) platform. This makes new algorithms and approaches to unusual types of data (such as images, videos and sounds) more accessible and deployable.”

Immuta

College Park, Maryland
www.immuta.com
“Immuta offers a dedicated data access and management platform for the development of machine learning and other advanced analytics, and the automation of policy enforcement,” Peter Krensky and Jim Hare write. “The product serves as a control layer to rapidly connect and control access between myriad data sources and the heterogeneous array of data science tools without the need to move or copy data. This approach addresses the market expectation that platforms supporting data science will be highly flexible and extensible to the data portfolio and toolkit of a user’s choosing.”

Indico

Boston, Massachusetts
www.indico.io
“Indico offers a group of products with a highly accessible set of functionality for exploring and modeling unstructured data and automating processes,” according to Peter Krensky and Austin Kronz. “The offering can be described as a citizen data science toolkit for applying deep learning to text, images and document-based data. Indico’s approach makes deep learning a practical solution for subject matter experts (SMEs) facing unstructured content challenges. This is ambitious and exciting, as both deep learning and unstructured content analytics are areas where even expert data scientists are still climbing the learning curve.”

Octopai

Rosh HaAyin, Israel & New York, New York
www.octopai.com
“Octopai solves a foundational problem for data-driven organizations — enabling data science teams and citizen data scientists to quickly find the data, establish trust in data sources and achieve transparency of data lineage through automation,” explains Svetlana Sicular. “It connects the dots of complex data pipelines by using machine learning and pattern analysis to determine the relationships among different data elements, the context in which the data was created, and the data’s prior uses and transformations. Such access to more diverse, transparent and trustworthy data leads to better quality analytics and machine learning.”

ParallelM

Tel Aviv, Israel & Sunnyvale, California
www.parallelm.com
“ParallelM is one of the first software platforms principally focused on the data science operationalization process,” Erick Brethenoux writes. “The focus of data science teams has traditionally been on developing analytical assets, while dealing with the operationalization of these assets has been an afterthought. Deploying analytical assets within operational processes in a repeatable, manageable, secure and traceable manner requires more than a set of APIs and a cloud service; a model that has been scored (executed) has not necessarily been managed. ParallelM’s success and the general development of operationalization functionality within platforms will be an indicator of the success of an entire generation of data scientists.”

 Source: Information Management

Gartner: US government agencies falling behind digital businesses in other industries

A Gartner survey of more than 500 government CIOs shows that government agencies are falling behind other industries when it comes to planned investments in digital business initiatives. Just 17% of government CIOs say they’ll be increasing their investments, compared to 34% of CIOs in other industries.

What’s holding government agencies back? While Gartner notes that their CIOs demonstrate a clear vision for the potential of digital government and emerging technologies, almost half of those surveyed (45%) say they lack the IT and business resources required to execute. Other common barriers include lack of funding (39%), as well as a challenge organizations across all industries struggle with: culture and resistance to change (37%).

Another key challenge is the ability to scale digital initiatives, where government agencies lag by 5% against all other industries. To catch up, government CIOs see automation as a potential tool. This aligns with respondents’ views on 'game-changing' technologies for government. The top five in order are:

• Artificial intelligence (AI) and machine learning (27%)
• Data analytics, including predictive analytics (22%)
• Cloud (19%)
• Internet of Things (7%)
• Mobile, including 5G (6%)

Of the more than 500 government respondents in Gartner’s survey, 10% have already deployed an AI solution, 39% say they plan to deploy one within the next one to two years, and 36% intend to use AI to enable automation, scale of digital initiatives, and reallocation of human resources within the next two to three years.

Investing today for tomorrow's success

When it comes to increased investment this year (2019), BI and data analytics (43%), cyber and information security (43%), and cloud services and solutions (39%) top the tech funding list.

As previous and current digital government initiatives start to take hold, CIOs are seeing moderate improvements in their ability to meet the increasing demands and expectations of citizens. 65% of CIOs say that their current digital government investments are already paying off. A great example of this is the U.S. Department of Housing and Urban Development’s use of BI and data analytics to modernize its Grants Dashboard.

Despite budget and cultural change challenges typically associated with digital government initiatives, make no mistake: many agencies are making great strides and are now competing or leading compared to other organizations and industries.

There’s never been a better time to invest in game changing technologies to both quickly catch up, and potentially take the lead.

Author: Rick Nelson

Source: Microstrategy

Getting Your Machine Learning Model To Production: Why Does It Take So Long?

A Gentle Guide to the complexities of model deployment, and integrating with the enterprise application and data pipeline. What the Data Scientist, Data Engineer, ML Engineer, and ML Ops do, in Plain English.

Let’s say we’ve identified a high-impact business problem at our company, built an ML (machine learning) model to tackle it, trained it, and are happy with the prediction results. This was a hard problem to crack that required much research and experimentation. So we’re excited about finally being able to use the model to solve our user’s problem!

However, what we’ll soon discover is that building the model itself is only the tip of the iceberg. The bulk of the hard work to actually put this model into production is still ahead of us. I’ve found that this second stage could take even up to 90% of the time and effort for the project.

So what does this stage comprise of? And why is it that it takes so much time? That is the focus of this article.

Over several articles, my goal is to explore various facets of an organization’s ML journey as it goes all the way from deploying its first ML model to setting up an agile development and deployment process for rapid experimentation and delivery of ML projects. In order to understand what needs to be done in the second stage, let’s first see what gets delivered at the end of the first stage.

What does the Model Building and Training phase deliver?

Models are typically built and trained by the Data Science team. When it is ready, we have model code in Jupyter notebooks along with trained weights.

• It is often trained using a static snapshot of the dataset, perhaps in a CSV or Excel file.
• The snapshot was probably a subset of the full dataset.
• Training is run on a developer’s local laptop, or perhaps on a VM in the cloud

In other words, the development of the model is fairly standalone and isolated from the company’s application and data pipelines.

What does “Production” mean?

When a model is put into production, it operates in two modes:

• Real-time Inference — perform online predictions on new input data, on a single sample at a time
• Retraining — for offline retraining of the model nightly or weekly, with a current refreshed dataset

The requirements and tasks involved for these two modes are quite different. This means that the model gets put into two production environments:

• A Serving environment for performing Inference and serving predictions
• A Training environment for retraining

 

Real-time Inference and Retraining in Production (Source: Author)

Real-time Inference is what most people would have in mind when they think of “production”. But there are also many use cases that do Batch Inference instead of Real-time.

• Batch Inference — perform offline predictions nightly or weekly, on a full dataset

 

Batch Inference and Retraining in Production (Source: Author)

For each of these modes separately, the model now needs to be integrated with the company’s production systems — business application, data pipeline, and deployment infrastructure. Let’s unpack each of these areas to see what they entail.

We’ll start by focusing on Real-time Inference, and after that, we’ll examine the Batch cases (Retraining and Batch Inference). Some of the complexities that come up are unique to ML, but many are standard software engineering challenges.

Inference — Application Integration

A model usually is not an independent entity. It is part of a business application for end users eg. a recommender model for an e-commerce site. The model needs to be integrated with the interaction flow and business logic of the application.

The application might get its input from the end-user via a UI and pass it to the model. Alternately, it might get its input from an API endpoint, or from a streaming data system. For instance, a fraud detection algorithm that approves credit card transactions might process transaction input from a Kafka topic.

Similarly, the output of the model gets consumed by the application. It might be presented back to the user in the UI, or the application might use the model’s predictions to make some decisions as part of its business logic.

Inter-process communication between the model and the application needs to be built. For example, we might deploy the model as its own service accessed via an API call. Alternately, if the application is also written in the same programming language (eg. Python), it could just make a local function call to the model code.

This work is usually done by the Application Developer working closely with the Data Scientist. As with any integration between modules in a software development project, this requires collaboration to ensure that assumptions about the formats and semantics of the data flowing back and forth are consistent on both sides. We all know the kinds of issues that can crop up. eg. If the model expects a numeric ‘quantity’ field to be non-negative, will the application do the validation before passing it to the model? Or is the model expected to perform that check? In what format is the application passing dates and does the model expect the same format?

 

Real-time Inference Lifecycle (Source: Author)

Inference — Data Integration

The model can no longer rely on a static dataset that contains all the features it needs to make its predictions. It needs to fetch ‘live’ data from the organization’s data stores.

These features might reside in transactional data sources (eg. a SQL or NoSQL database), or they might be in semi-structured or unstructured datasets like log files or text documents. Perhaps some features are fetched by calling an API, either an internal microservice or application (eg. SAP) or an external third-party endpoint.

If any of this data isn’t in the right place or in the right format, some ETL (Extract, Transform, Load) jobs may have to be built to pre-fetch the data to the store that the application will use.

Dealing with all the data integration issues can be a major undertaking. For instance:

• Access requirements — how do you connect to each data source, and what are its security and access control policies?
• Handle errors — what if the request times out, or the system is down?
• Match latencies — how long does a query to the data source take, versus how quickly do we need to respond to the user?
• Sensitive data — Is there personally identifiable information that has to be masked or anonymized.
• Decryption — does data need to decrypted before the model can use it?
• Internationalization — can the model handle the necessary character encodings and number/date formats?
• and many more…

This tooling gets built by a Data Engineer. For this phase as well, they would interact with the Data Scientist to ensure that the assumptions are consistent and the integration goes smoothly. eg. Is the data cleaning and pre-processing done by the model enough, or do any more transformations have to be built?

Inference — Deployment

It is now time to deploy the model to the production environment. All the factors that one considers with any software deployment come up:

• Model Hosting — on a mobile app? In an on-premise data center or on the cloud? On an embedded device?
• Model Packaging — what dependent software and ML libraries does it need? These are typically different from your regular application libraries.
• Co-location — will the model be co-located with the application? Or as an external service?
• Model Configuration settings — how will they be maintained and updated?
• System resources required — CPU, RAM, disk, and most importantly GPU, since that may need specialized hardware.
• Non-functional requirements — volume and throughput of request traffic? What is the expected response time and latency?
• Auto-Scaling — what kind of infrastructure is required to support it?
• Containerization — does it need to be packaged into a Docker container? How will container orchestration and resource scheduling be done?
• Security requirements — credentials to be stored, private keys to be managed in order to access data?
• Cloud Services — if deploying to the cloud, is integration with any cloud services required eg. (Amazon Web Services) AWS S3? What about AWS access control privileges?
• Automated deployment tooling — to provision, deploy and configure the infrastructure and install the software.
• CI/CD — automated unit or integration tests to integrate with the organization’s CI/CD pipeline.

The ML Engineer is responsible for implementing this phase and deploying the application into production. Finally, you’re able to put the application in front of the customer, which is a significant milestone!

However, it is not yet time to sit back and relax 😃. Now begins the ML Ops task of monitoring the application to make sure that it continues to perform optimally in production.

Inference — Monitoring

The goal of monitoring is to check that your model continues to make correct predictions in production, with live customer data, as it did during development. It is quite possible that your metrics will not be as good.

In addition, you need to monitor all the standard DevOps application metrics just like you would for any application — latency, response time, throughput as well as system metrics like CPU utilization, RAM, etc. You would run the normal health checks to ensure uptime and stability of the application.

Equally importantly, monitoring needs to be an ongoing process, because there is every chance that your model’s evaluation metrics will deteriorate with time. Compare your evaluation metrics to past metrics to check that there is no deviation from historical trends.

This can happen because of data drift.

Inference — Data Validation

As time goes on, your data will evolve and change — new data sources may get added, new feature values will get collected, new customers will input data with different values than before. This means that the distribution of your data could change.

So validating your model with current data needs to be an ongoing activity. It is not enough to look only at evaluation metrics for the global dataset. You should evaluate metrics for different slices and segments of your data as well. It is very likely that as your business evolves and as customer demographics, preferences, and behavior change, your data segments will also change.

The data assumptions that were made when the model was first built may no longer hold true. To account for this, your model needs to evolve as well. The data cleaning and pre-processing that the model does might also need to be updated.

And that brings us to the second production mode — that of Batch Retraining on a regular basis so that the model continues to learn from fresh data. Let’s look at the tasks required to set up Batch Retraining in production, starting with the development model.

 

Retraining Lifecycle (Source: Author)

Retraining — Data Integration

When we discussed Data Integration for Inference, it involved fetching a single sample of the latest ‘live’ data. On the other hand, during Retraining, we need to fetch a full dataset of historical data. Also, this Retraining happens in batch mode, say every night or every week.

Historical doesn’t necessarily mean “old and outdated” data — it could include all of the data gathered until yesterday, for instance.

This dataset would typically reside in an organization’s analytics stores, such as a data warehouse or data lake. If some data isn’t present there, you might need to build additional ETL jobs to transfer that data into the warehouse in the required format.

Retraining — Application Integration

Since we’re only retraining the model by itself, the whole application is not involved. So no Application Integration work is needed.

Retraining — Deployment

Retraining is likely to happen with a massive amount of data, probably far larger than what was used during development.

You will need to figure out the hardware infrastructure needed to train the model — what are its GPU and RAM requirements? Since training needs to complete in a reasonable amount of time, it will need to be distributed across many nodes in a cluster, so that training happens in parallel. Each node will need to be provisioned and managed by a Resource Scheduler so that hardware resources can be efficiently allocated to each training process.

The setup will also need to ensure that these large data volumes can be efficiently transferred to all the nodes on which the training is being executed.

And before we wrap up, let’s look at our third production use case — the Batch Inference scenario.

Batch Inference

Often, the Inference does not have to run ‘live’ in real-time for a single data item at a time. There are many use cases for which it can be run as a batch job, where the output results for a large set of data samples are pre-computed and cached.

The pre-computed results can then be used in different ways depending on the use case. eg.

• They could be stored in the data warehouse for reporting or for interactive analysis by business analysts.
• They could be cached and displayed by the application to the user when they log in next.
• Or they could be cached and used as input features by another downstream application.

For instance, a model that predicts the likelihood of customer churn (ie. they stop buying from you) can be run every week or every night. The results could then be used to run a special promotion for all customers who are classified as high risks. Or they could be presented with an offer when they next visit the site.

A Batch Inference model might be deployed as part of a workflow with a network of applications. Each application is executed after its dependencies have completed.

Many of the same application and data integration issues that come up with Real-time Inference also apply here. On the other hand, Batch Inference does not have the same response-time and latency demands. But, it does have high throughput requirements as it deals with enormous data volumes.

Conclusion

As we have just seen, there are many challenges and a significant amount of work to put a model in production. Even after the Data Scientists ready a trained model, there are many roles in an organization that all come together to eventually bring it to your customers and to keep it humming month after month. Only then does the organization truly get the benefit of harnessing machine learning.

We’ve now seen the complexity of building and training a real-world model, and then putting it into production. In the next article, we’ll take a look at how the leading-edge tech companies have addressed these problems to churn out ML applications rapidly and smoothly.

And finally, if you liked this article, you might also enjoy my other series on Transformers, Audio Deep Learning, and Geolocation Machine Learning.

Author: Ketan Doshi

Source: Towards Data Science

Green AI: how AI poses both problems and solutions regarding climate change

AI and ML are making a significant contribution to climate change. Developers can help reverse the trend with best practices and tools to measure carbon efficiency.

The growth of computationally intensive technologies such as machine learning incurs a high carbon footprint and is contributing to climate change. Alongside that rapid growth is an expanding portfolio of green AI tools and techniques to help offset carbon usage and provide a more sustainable path forward.

The cost to the environment is high, according to research published last month by Microsoft and the Allen Institute for AI, with co-authors from Hebrew University, Carnegie Mellon University and Hugging Face, an AI community. The study extrapolated data to show that one training instance for a single 6 billion parameter transformer ML model -- a large language model -- is the CO2 equivalent to burning all the coal in a large railroad car, according to Will Buchanan, product manager for Azure machine learning at Microsoft, Green Software Foundation member and co-author of the study.

In the past, code was optimized in embedded systems that are constrained by limited resources such as those seen in phones, refrigerators or satellites, said Abhijit Sunil, analyst at Forrester Research. However, emerging technologies such as AI and ML aren't subject to those limitations, he said.

"When we have seemingly unlimited resources, what took precedence was to make as much code as possible," Sunil said.

Is AI the right tool for the job?

Green AI, or the process of making AI development more sustainable, is emerging as a possible solution to the problem of power-hungry algorithms. "It is all about reducing the hidden costs of the development of the technology itself," Buchanan said.

A starting point for any developer is to ask if AI is the right tool for the job and to be clear on why machine learning is being deployed in the first place, said Abhishek Gupta, founder and principal researcher at the Montreal AI Ethics Institute and chair of the Green Software Foundation's standards working group.

"You don't always need machine learning to solve a problem," Gupta said.

Developers should also consider conducting a cost-benefit analysis when deploying ML, Gupta said. For example, if the use of ML increases a platform's satisfaction rate from 95% to 96%, that might not be worth the additional cost to the environment, he said.

Choose a carbon-friendly region

Once a developer has decided to use AI, then choosing to deploy a model in a carbon-friendly region can have the largest effect on operational emissions, reducing the Software Carbon Intensity rate by about 75%, Buchanan said.

"It's the most impactful lever that any developer today can use," Buchanan said.

Gupta provided the following example: Instead of running a job in the Midwestern U.S., where electricity is primarily obtained from fossil fuels, developers can choose to run it in Quebec, which garners more than 90% of its electricity from hydro.

Companies will also have to consider other factors beyond energy type when deciding where an ML job should run. In April 2021, Google Cloud introduced its green region picker, which helps companies evaluate costs, latency and carbon footprint when choosing where to operate. But tools like these aren't readily available from all cloud providers, Buchanan said.

To address the issue, the Green Software Foundation is working on a new tool called Carbon Aware SDK, which will recommend the best region to spin up resources, he said. An alpha version should be available within the next couple of months.

Other ways to be green

If the only available computer is in a dirty electricity region, developers could use a federated learning-style deployment where training happens in a distributed fashion across all devices that exist in an electricity regime, Gupta said. But federated learning might not work for all workloads, such as those that must adhere to legal privacy considerations.

Another option is for developers to use tinyML, which shrinks machine learning models through quantization, knowledge distillation and other approaches, Gupta said. The goal is to minimize the models so that they can be deployed in a more resource-efficient way, such as on edge devices, he said. But as these models deliver limited intelligence, they might not be suited for complex use cases.

Sparse and shallow trees -- tree-based models partitioned into a small number of regions with sparse features -- can also provide the same results at less cost, Buchanan said. Developers can easily define them with a set of parameters when choosing a neural net architecture, he said.

"There's an industrywide trend to think that bigger is always better, but our research is showing that you can push back on that and say specifically that you need to choose the right tool for the job," Buchanan said.

Consumption metrics could be the solution

The Green Software Foundation and other initiatives are making progress toward measuring and mitigating software's carbon footprint, Buchanan said.

For example, Microsoft made energy consumption metrics available last year within Azure Machine Learning, making it possible for developers to pinpoint their most energy-consuming jobs. The metrics are focused on power-hungry GPUs, which are faster than CPUs but can consume more than 10 times the energy. Often used for running AI models, GPUs are typically the biggest culprit when it comes to power consumption, Buchanan said.

However, what's still needed is more interoperable tooling, Buchanan said, referring to the piecemeal green AI tools that are currently available. "The Green Software Foundation is doing one piece," he said, "but I think cloud providers need to make concerted investments to become more energy efficient."

Ultimately, the goal is to trigger behavior change so that green AI practices become the norm, according to Gupta. "We're not just doing this for accounting purposes," he said.

Author: Stephanie Glen

Source: TechTarget

Flexibel en schaalbaar managen van big data

Data-infrastructuur is het belangrijkste orgaan voor het creëren en leveren van goede bedrijfsinzichten . Om te profiteren van de diversiteit aan data die voor handen zijn en om de data-architectuur te moderniseren, zetten veel organisaties Hadoop in. Een Hadoop-gebaseerde omgeving is flexibel en schaalbaar in het managen van big data. Wat is de impact van Hadoop? De Aberdeen Group onderzocht de impact van Hadoop op data, mensen en de performance van bedrijven.

Nieuwe data uit verschillende bronnen

Er moet veel data opgevangen, verplaatst, opgeslagen en gearchiveerd worden. Maar bedrijven krijgen nu inzichten vanuit verborgen data buiten de traditionele gestructureerde transactiegegevens. Denk hierbij aan: e-mails, social data, multimedia, GPS-informatie en sensor-informatie. Naast nieuwe databronnen hebben we ook een grote hoeveelheid nieuwe technologieën gekregen om al deze data te beheren en te benutten. Al deze informatie en technologieën zorgen voor een verschuiving binnen big data; van probleem naar kans.

Wat zijn de voordelen van deze gele olifant (Hadoop)?

Een grote voorloper van deze big data-kans is de data architectuur Hadoop. Uit dit onderzoek komt naar voren dat bedrijven die Hadoop gebruiken meer gedreven zijn om gebruik te maken van ongestructureerde en semigestructureerd data. Een andere belangrijke trend is dat de mindset van bedrijven verschuift, ze zien data als een strategische aanwinst en als een belangrijk onderdeel van de organisatie.

De behoefte aan gebruikersbevoegdheid en gebruikerstevredenheid is een reden waarom bedrijven kiezen voor Hadoop. Daarnaast heeft een Hadoop-gebaseerde architectuur twee voordelen met betrekking tot eindgebruikers:

1. Data-flexibiliteit – Alle data onder één dak, wat zorgt voor een hogere kwaliteit en usability.
2. Data-elasticiteit – De architectuur is significant flexibeler in het toevoegen van nieuwe databronnen.

Wat is de impact van Hadoop op uw organisatie?

Wat kunt u nog meer met Hadoop en hoe kunt u deze data-architectuur het beste inzetten binnen uw databronnen? Lees in dit rapport hoe u nog meer tijd kunt besparen in het analyseren van data en uiteindelijk meer winst kunt behalen door het inzetten van Hadoop.

Bron: Analyticstoday

Data scientists are known for their unique skill sets. While thousands of compelling articles have been written about what a data scientist does, most of these articles fall short in examining what happens after you’ve hired a new data scientist to your team. 

The onboarding process for your data scientist should be based on the skills and areas of improvement you’ve identified for the tasks you want them to complete. Here’s how we do it at Elicit.

We’ve all seen the data scientist Venn diagrams over the past few years, which includes three high-level types of skills: programming, statistics/modeling, and domain expertise. Some even feature the ever-elusive “unicorn” at the center. 

While these diagrams provide us with a broad understanding of the skillset required for the role in general, they don’t have enough detail to differentiate data scientists and their roles inside a specific organization. This can lead to poor hires and poor onboarding experiences.

If the root of what a data scientist does and is capable of is not well understood, then both parties are in for a bad experience. Near the end of 2016, Anand Ramanathan wrote a post that really stuck with me called //medium.com/@anandr42/the-data-science-delusion-7759f4eaac8e" style="box-sizing:border-box;background-color:transparent;color:rgb(204, 51, 51);text-decoration:none">The Data Science Delusion. In it, Ramanathan talks about how within each layer of the data science Venn diagram there are degrees of understanding and capability.

For example, Ramanathan breaks down the modeling aspect into four quadrants based on modeling difficulty and system complexity, explaining that not every data scientist has to be capable in all four quadrants—that different problems call for different solutions and different skillsets. 

For example, if I want to understand customer churn, I probably don’t need a deep learning solution. Conversely, if I’m trying to recognize images, a logistic regression probably isn’t going to help me much.

In short, you want your data scientist to be skilled in the specific areas that role will be responsible for within the context of your business.

Ramanathan’s article also made me reflect on our data science team here at Elicit. Anytime we want to solve a problem internally or with a client we use our "Geek Nerd Suit" framework to help us organize our thoughts.

Basically, it states that for any organization to run at optimal speed, the technology (Geek), analytics (Nerd), and business (Suit) functions must be collaborating and making decisions in lockstep. Upon closer inspection, the data science Venn diagram is actually comprised of Geek (programming), Nerd (statistics/modeling), and Suit (domain expertise) skills.

But those themes are too broad; they still lack the detail needed to differentiate the roles of a data scientist. And we’d heard this from our team internally: in a recent employee survey, the issue of career advancement, and more importantly, skills differentiation, cropped up from our data science team.

As a leadership team, we always knew the strengths and weaknesses of our team members, but for their own sense of career progression they were asking us to be more specific and transparent about them. This pushed us to go through the exercise of taking a closer look at our own evaluation techniques, and resulted in a list of specific competencies within the Geek, Nerd, and Suit themes. We now use these competencies both to assess new hires and to help them develop in their careers once they’ve joined us.

For example, under the Suit responsibilities we define a variety of competencies that, amongst other things, include adaptability, business acumen, and communication. Each competency then has explicit sets of criteria associated with them that illustrate a different level of mastery within that competency. 

We’ve established four levels of differentiation: “entry level,” “intermediate,” “advanced” and “senior.” To illustrate, here’s the distinction between “entry level” and “intermediate” for the Suit: Adaptability competency:

Entry Level:

• Analyzes both success and failures for clues to improvement.
• Maintains composure during client meetings, remaining cool under pressure and not becoming defensive, even when under criticism.

Intermediate:

• Experiments and perseveres to find solutions.
• Reads situations quickly.
• Swiftly learns new concepts, skills, and abilities when facing new problems.

And there are other specific criteria for the “advanced” and “senior” levels as well. 

This led us to four unique data science titles—Data Scientist I, II, and III, as well as Senior Data Scientist, with the latter title still being explored for further differentiation. 

The Geek Nerd Suit framework, and the definitions of the competencies within them, gives us clear, explicit criteria for assessing a new hire’s skillset in the three critical dimensions that are required for a data scientist to be successful.

In Part 2, I’ll discuss what we specifically do within the Geek Nerd Suit framework to onboard a new hire once they’ve joined us—how we begin to groom the elusive unicorn. 

Source: Information Management

Author: Liam Hanham

Helping Business Executives Understand Machine Learning

For data science teams to succeed, business leaders need to understand the importance of MLops, modelops, and the machine learning life cycle. Try these analogies and examples to cut through the jargon.

If you’re a data scientist or you work with machine learning (ML) models, you have tools to label data, technology environments to train models, and a fundamental understanding of MLops and modelops. If you have ML models running in production, you probably use ML monitoring to identify data drift and other model risks.

Data science teams use these essential ML practices and platforms to collaborate on model development, to configure infrastructure, to deploy ML models to different environments, and to maintain models at scale. Others who are seeking to increase the number of models in production, improve the quality of predictions, and reduce the costs in ML model maintenance will likely need these ML life cycle management tools, too.

Unfortunately, explaining these practices and tools to business stakeholders and budget decision-makers isn’t easy. It’s all technical jargon to leaders who want to understand the return on investment and business impact of machine learning and artificial intelligence investments and would prefer staying out of the technical and operational weeds.

Data scientists, developers, and technology leaders recognize that getting buy-in requires defining and simplifying the jargon so stakeholders understand the importance of key disciplines. Following up on a previous article about how to explain devops jargon to business executives, I thought I would write a similar one to clarify several critical ML practices that business leaders should understand.   

What is the machine learning life cycle?

As a developer or data scientist, you have an engineering process for taking new ideas from concept to delivering business value. That process includes defining the problem statement, developing and testing models, deploying models to production environments, monitoring models in production, and enabling maintenance and improvements. We call this a life cycle process, knowing that deployment is the first step to realizing the business value and that once in production, models aren’t static and will require ongoing support.

Business leaders may not understand the term life cycle. Many still perceive software development and data science work as one-time investments, which is one reason why many organizations suffer from tech debt and data quality issues.

Explaining the life cycle with technical terms about model development, training, deployment, and monitoring will make a business executive’s eyes glaze over. Marcus Merrell, vice president of technology strategy at Sauce Labs, suggests providing leaders with a real-world analogy.

“Machine learning is somewhat analogous to farming: The crops we know today are the ideal outcome of previous generations noticing patterns, experimenting with combinations, and sharing information with other farmers to create better variations using accumulated knowledge,” he says. “Machine learning is much the same process of observation, cascading conclusions, and compounding knowledge as your algorithm gets trained.”

What I like about this analogy is that it illustrates generative learning from one crop year to the next but can also factor in real-time adjustments that might occur during a growing season because of weather, supply chain, or other factors. Where possible, it may be beneficial to find analogies in your industry or a domain your business leaders understand.

What is MLops?

Most developers and data scientists think of MLops as the equivalent of devops for machine learning. Automating infrastructure, deployment, and other engineering processes improves collaborations and helps teams focus more energy on business objectives instead of manually performing technical tasks.

But all this is in the weeds for business executives who need a simple definition of MLops, especially when teams need budget for tools or time to establish best practices.

“MLops, or machine learning operations, is the practice of collaboration and communication between data science, IT, and the business to help manage the end-to-end life cycle of machine learning projects,” says Alon Gubkin, CTO and cofounder of Aporia. “MLops is about bringing together different teams and departments within an organization to ensure that machine learning models are deployed and maintained effectively.”

Thibaut Gourdel, technical product marketing manager at Talend, suggests adding some detail for the more data-driven business leaders. He says, “MLops promotes the use of agile software principles applied to ML projects, such as version control of data and models as well as continuous data validation, testing, and ML deployment to improve repeatability and reliability of models, in addition to your teams’ productivity.”

What is data drift?

Whenever you can use words that convey a picture, it’s much easier to connect the term with an example or a story. An executive understands what drift is from examples such as a boat drifting off course because of the wind, but they may struggle to translate it to the world of data, statistical distributions, and model accuracy.

“Data drift occurs when the data the model sees in production no longer resembles the historical data it was trained on,” says Krishnaram Kenthapadi, chief AI officer and scientist at Fiddler AI. “It can be abrupt, like the shopping behavior changes brought on by the COVID-19 pandemic. Regardless of how the drift occurs, it’s critical to identify these shifts quickly to maintain model accuracy and reduce business impact.”

Gubkin provides a second example of when data drift is a more gradual shift from the data the model was trained on. “Data drift is like a company’s products becoming less popular over time because consumer preferences have changed.”

David Talby, CTO of John Snow Labs, shared a generalized analogy. “Model drift happens when accuracy degrades due to the changing production environment in which it operates,” he says. “Much like a new car’s value declines the instant you drive it off the lot, a model does the same, as the predictable research environment it was trained on behaves differently in production. Regardless of how well it’s operating, a model will always need maintenance as the world around it changes.” 

The important message that data science leaders must convey is that because data isn’t static, models must be reviewed for accuracy and be retrained on more recent and relevant data.

What is ML monitoring?

How does a manufacturer measure quality before their products are boxed and shipped to retailers and customers? Manufacturers use different tools to identify defects, including when an assembly line is beginning to show deviations from acceptable output quality. If we think of an ML model as a small manufacturing plant producing forecasts, then it makes sense that data science teams need ML monitoring tools to check for performance and quality issues. Katie Roberts, data science solution architect at Neo4j, says, “ML monitoring is a set of techniques used during production to detect issues that may negatively impact model performance, resulting in poor-quality insights.”

Manufacturing and quality control is an easy analogy, and here are two recommendations to provide ML model monitoring specifics: “As companies accelerate investment in AI/ML initiatives, AI models will increase drastically from tens to thousands. Each needs to be stored securely and monitored continuously to ensure accuracy,” says Hillary Ashton, chief product officer at Teradata. 

What is modelops?

MLops focuses on multidisciplinary teams collaborating on developing, deploying, and maintaining models. But how should leaders decide what models to invest in, which ones require maintenance, and where to create transparency around the costs and benefits of artificial intelligence and machine learning?

These are governance concerns and part of what modelops practices and platforms aim to address. Business leaders want modelops but won’t fully understand the need and what it delivers until its partially implemented.

That’s a problem, especially for enterprises that seek investment in modelops platforms. Nitin Rakesh, CEO and managing director of Mphasis suggests explaining modelops this way. “By focusing on modelops, organizations can ensure machine learning models are deployed and maintained to maximize value and ensure governance for different versions.“

Ashton suggests including one example practice. “Modelops allows data scientists to identify and remediate data quality risks, automatically detect when models degrade, and schedule model retraining,” she says.

There are still many new ML and AI capabilities, algorithms, and technologies with confusing jargon that will seep into a business leader’s vocabulary. When data specialists and technologists take time to explain the terminology in language business leaders understand, they are more likely to get collaborative support and buy-in for new investments.

Author: Isaac Sacolick

Soruce: InfoWorld

Computers en apparaten die met ons meedenken zijn al lang geen sciencefiction meer. Artificial intelligence (AI) is terug te vinden in wasmachines die hun programma aanpassen aan de hoeveelheid was en computerspellen die zich aanpassen aan het niveau van de spelers. Hoe kunnen computers mensen helpen slimmer te beslissen? Deze uitgebreide whitepaper beschrijft welke modellen in het analyseplatform HPE IDOL worden toegepast.

Mathematische modellen zorgen voor menselijke maat

Processors kunnen in een oogwenk een berekening uitvoeren waar mensen weken tot maanden mee bezig zouden zijn. Daarom zijn computers betere schakers dan mensen, maar slechter in poker waarin de menselijke maat een grotere rol speelt. Hoe zorgt een zoek- en analyseplatform ervoor dat er meer ‘mens’ in de analyse terechtkomt? Dat wordt gerealiseerd door gebruik te maken van verschillende mathematische modellen.

Analyses voor tekst, geluid, beeld en gezichten

De kunst is om uit data actiegerichte informatie te verkrijgen. Dat lukt door patroonherkenning in te zetten op verschillende datasets. Daarnaast spelen classificatie, clustering en analyse een grote rol bij het verkrijgen van de juiste inzichten. Niet alleen teksten worden geanalyseerd, steeds vaker worden ook geluidsbestanden en beelden, objecten en gezichten geanalyseerd.

Artificial intelligence helpt de mens

De whitepaper beschrijft uitvoerig hoe patronen worden gevonden in tekst, audio en beelden. Hoe snapt een computer dat de video die hij analyseert over een mens gaat? Hoe wordt van platte beelden een geometrisch 3d-beeld gemaakt en hoe beslist een computer wat hij ziet? Denk bijvoorbeeld aan een geautomatiseerd seintje naar de controlekamer als het te druk is op een tribune of een file ontstaat. Hoe helpen theoretische modellen computers als mensen waarnemen en onze beslissingen ondersteunen? Dat en meer leest u in de whitepaper Augmented intelligence Helping humans make smarter decisions. Zie hiervoor AnalyticsToday

Analyticstoday.nl, 12 oktober 2016

How algorithms mislead the human brain in social media - Part 1

Consider Andy, who is worried about contracting COVID-19. Unable to read all the articles he sees on it, he relies on trusted friends for tips. When one opines on Facebook that pandemic fears are overblown, Andy dismisses the idea at first. But then the hotel where he works closes its doors, and with his job at risk, Andy starts wondering how serious the threat from the new virus really is. No one he knows has died, after all. A colleague posts an article about the COVID “scare” having been created by Big Pharma in collusion with corrupt politicians, which jibes with Andy's distrust of government. His Web search quickly takes him to articles claiming that COVID-19 is no worse than the flu. Andy joins an online group of people who have been or fear being laid off and soon finds himself asking, like many of them, “What pandemic?” When he learns that several of his new friends are planning to attend a rally demanding an end to lockdowns, he decides to join them. Almost no one at the massive protest, including him, wears a mask. When his sister asks about the rally, Andy shares the conviction that has now become part of his identity: COVID is a hoax.

This example illustrates a minefield of cognitive biases. We prefer information from people we trust, our in-group. We pay attention to and are more likely to share information about risks—for Andy, the risk of losing his job. We search for and remember things that fit well with what we already know and understand. These biases are products of our evolutionary past, and for tens of thousands of years, they served us well. People who behaved in accordance with them—for example, by staying away from the overgrown pond bank where someone said there was a viper—were more likely to survive than those who did not.

Modern technologies are amplifying these biases in harmful ways, however. Search engines direct Andy to sites that inflame his suspicions, and social media connects him with like-minded people, feeding his fears. Making matters worse, bots—automated social media accounts that impersonate humans—enable misguided or malevolent actors to take advantage of his vulnerabilities.

Compounding the problem is the proliferation of online information. Viewing and producing blogs, videos, tweets and other units of information called memes has become so cheap and easy that the information marketplace is inundated. Unable to process all this material, we let our cognitive biases decide what we should pay attention to. These mental shortcuts influence which information we search for, comprehend, remember and repeat to a harmful extent.

The need to understand these cognitive vulnerabilities and how algorithms use or manipulate them has become urgent. At the University of Warwick in England and at Indiana University Bloomington's Observatory on Social Media (OSoMe, pronounced “awesome”), our teams are using cognitive experiments, simulations, data mining and artificial intelligence to comprehend the cognitive vulnerabilities of social media users. Insights from psychological studies on the evolution of information conducted at Warwick inform the computer models developed at Indiana, and vice versa. We are also developing analytical and machine-learning aids to fight social media manipulation. Some of these tools are already being used by journalists, civil-society organizations and individuals to detect inauthentic actors, map the spread of false narratives and foster news literacy.

Information Overload

The glut of information has generated intense competition for people's attention. As Nobel Prize–winning economist and psychologist Herbert A. Simon noted, “What information consumes is rather obvious: it consumes the attention of its recipients.” One of the first consequences of the so-called attention economy is the loss of high-quality information. The OSoMe team demonstrated this result with a set of simple simulations. It represented users of social media such as Andy, called agents, as nodes in a network of online acquaintances. At each time step in the simulation, an agent may either create a meme or reshare one that he or she sees in a news feed. To mimic limited attention, agents are allowed to view only a certain number of items near the top of their news feeds.

Running this simulation over many time steps, Lilian Weng of OSoMe found that as agents' attention became increasingly limited, the propagation of memes came to reflect the power-law distribution of actual social media: the probability that a meme would be shared a given number of times was roughly an inverse power of that number. For example, the likelihood of a meme being shared three times was approximately nine times less than that of its being shared once.

This winner-take-all popularity pattern of memes, in which most are barely noticed while a few spread widely, could not be explained by some of them being more catchy or somehow more valuable: the memes in this simulated world had no intrinsic quality. Virality resulted purely from the statistical consequences of information proliferation in a social network of agents with limited attention. Even when agents preferentially shared memes of higher quality, researcher Xiaoyan Qiu, then at OSoMe, observed little improvement in the overall quality of those shared the most. Our models revealed that even when we want to see and share high-quality information, our inability to view everything in our news feeds inevitably leads us to share things that are partly or completely untrue.

Cognitive biases greatly worsen the problem. In a set of groundbreaking studies in 1932, psychologist Frederic Bartlett told volunteers a Native American legend about a young man who hears war cries and, pursuing them, enters a dreamlike battle that eventually leads to his real death. Bartlett asked the volunteers, who were non-Native, to recall the rather confusing story at increasing intervals, from minutes to years later. He found that as time passed, the rememberers tended to distort the tale's culturally unfamiliar parts such that they were either lost to memory or transformed into more familiar things. We now know that our minds do this all the time: they adjust our understanding of new information so that it fits in with what we already know. One consequence of this so-called confirmation bias is that people often seek out, recall and understand information that best confirms what they already believe.

This tendency is extremely difficult to correct. Experiments consistently show that even when people encounter balanced information containing views from differing perspectives, they tend to find supporting evidence for what they already believe. And when people with divergent beliefs about emotionally charged issues such as climate change are shown the same information on these topics, they become even more committed to their original positions.

Making matters worse, search engines and social media platforms provide personalized recommendations based on the vast amounts of data they have about users' past preferences. They prioritize information in our feeds that we are most likely to agree with—no matter how fringe—and shield us from information that might change our minds. This makes us easy targets for polarization. Nir Grinberg and his co-workers at Northeastern University recently showed that conservatives in the U.S. are more receptive to misinformation. But our own analysis of consumption of low-quality information on Twitter shows that the vulnerability applies to both sides of the political spectrum, and no one can fully avoid it. Even our ability to detect online manipulation is affected by our political bias, though not symmetrically: Republican users are more likely to mistake bots promoting conservative ideas for humans, whereas Democrats are more likely to mistake conservative human users for bots.

Social Herding

In New York City in August 2019, people began running away from what sounded like gunshots. Others followed, some shouting, “Shooter!” Only later did they learn that the blasts came from a backfiring motorcycle. In such a situation, it may pay to run first and ask questions later. In the absence of clear signals, our brains use information about the crowd to infer appropriate actions, similar to the behavior of schooling fish and flocking birds.

Such social conformity is pervasive. In a fascinating 2006 study involving 14,000 Web-based volunteers, Matthew Salganik, then at Columbia University, and his colleagues found that when people can see what music others are downloading, they end up downloading similar songs. Moreover, when people were isolated into “social” groups, in which they could see the preferences of others in their circle but had no information about outsiders, the choices of individual groups rapidly diverged. But the preferences of “nonsocial” groups, where no one knew about others' choices, stayed relatively stable. In other words, social groups create a pressure toward conformity so powerful that it can overcome individual preferences, and by amplifying random early differences, it can cause segregated groups to diverge to extremes.

Social media follows a similar dynamic. We confuse popularity with quality and end up copying the behavior we observe. Experiments on Twitter by Bjarke Mønsted and his colleagues at the Technical University of Denmark and the University of Southern California indicate that information is transmitted via “complex contagion”: when we are repeatedly exposed to an idea, typically from many sources, we are more likely to adopt and reshare it. This social bias is further amplified by what psychologists call the “mere exposure” effect: when people are repeatedly exposed to the same stimuli, such as certain faces, they grow to like those stimuli more than those they have encountered less often.

Such biases translate into an irresistible urge to pay attention to information that is going viral—if everybody else is talking about it, it must be important. In addition to showing us items that conform with our views, social media platforms such as Facebook, Twitter, YouTube and Instagram place popular content at the top of our screens and show us how many people have liked and shared something. Few of us realize that these cues do not provide independent assessments of quality.

In fact, programmers who design the algorithms for ranking memes on social media assume that the “wisdom of crowds” will quickly identify high-quality items; they use popularity as a proxy for quality. Our analysis of vast amounts of anonymous data about clicks shows that all platforms—social media, search engines and news sites—preferentially serve up information from a narrow subset of popular sources.

To understand why, we modeled how they combine signals for quality and popularity in their rankings. In this model, agents with limited attention—those who see only a given number of items at the top of their news feeds—are also more likely to click on memes ranked higher by the platform. Each item has intrinsic quality, as well as a level of popularity determined by how many times it has been clicked on. Another variable tracks the extent to which the ranking relies on popularity rather than quality. Simulations of this model reveal that such algorithmic bias typically suppresses the quality of memes even in the absence of human bias. Even when we want to share the best information, the algorithms end up misleading us.

Want to continue reading? You can find part 2 of this article herehere

Authors: Filippo Menczer

Source: Scientific American

How algorithms mislead the human brain in social media - Part 2

If you haven't read part 1 of this article yet, be sure to check it out here.

Echo Chambers

Most of us do not believe we follow the herd. But our confirmation bias leads us to follow others who are like us, a dynamic that is sometimes referred to as homophily—a tendency for like-minded people to connect with one another. Social media amplifies homophily by allowing users to alter their social network structures through following, unfriending, and so on. The result is that people become segregated into large, dense and increasingly misinformed communities commonly described as echo chambers.

At OSoMe, we explored the emergence of online echo chambers through another simulation, EchoDemo. In this model, each agent has a political opinion represented by a number ranging from −1 (say, liberal) to +1 (conservative). These inclinations are reflected in agents' posts. Agents are also influenced by the opinions they see in their news feeds, and they can unfollow users with dissimilar opinions. Starting with random initial networks and opinions, we found that the combination of social influence and unfollowing greatly accelerates the formation of polarized and segregated communities.

Indeed, the political echo chambers on Twitter are so extreme that individual users' political leanings can be predicted with high accuracy: you have the same opinions as the majority of your connections. This chambered structure efficiently spreads information within a community while insulating that community from other groups. In 2014 our research group was targeted by a disinformation campaign claiming that we were part of a politically motivated effort to suppress free speech. This false charge spread virally mostly in the conservative echo chamber, whereas debunking articles by fact-checkers were found mainly in the liberal community. Sadly, such segregation of fake news items from their fact-check reports is the norm.

Social media can also increase our negativity. In a recent laboratory study, Robert Jagiello, also at Warwick, found that socially shared information not only bolsters our biases but also becomes more resilient to correction. He investigated how information is passed from person to person in a so-called social diffusion chain. In the experiment, the first person in the chain read a set of articles about either nuclear power or food additives. The articles were designed to be balanced, containing as much positive information (for example, about less carbon pollution or longer-lasting food) as negative information (such as risk of meltdown or possible harm to health).

The first person in the social diffusion chain told the next person about the articles, the second told the third, and so on. We observed an overall increase in the amount of negative information as it passed along the chain—known as the social amplification of risk. Moreover, work by Danielle J. Navarro and her colleagues at the University of New South Wales in Australia found that information in social diffusion chains is most susceptible to distortion by individuals with the most extreme biases.

Even worse, social diffusion also makes negative information more “sticky.” When Jagiello subsequently exposed people in the social diffusion chains to the original, balanced information—that is, the news that the first person in the chain had seen—the balanced information did little to reduce individuals' negative attitudes. The information that had passed through people not only had become more negative but also was more resistant to updating.

A 2015 study by OSoMe researchers Emilio Ferrara and Zeyao Yang analyzed empirical data about such “emotional contagion” on Twitter and found that people overexposed to negative content tend to then share negative posts, whereas those overexposed to positive content tend to share more positive posts. Because negative content spreads faster than positive content, it is easy to manipulate emotions by creating narratives that trigger negative responses such as fear and anxiety. Ferrara, now at the University of Southern California, and his colleagues at the Bruno Kessler Foundation in Italy have shown that during Spain's 2017 referendum on Catalan independence, social bots were leveraged to retweet violent and inflammatory narratives, increasing their exposure and exacerbating social conflict.

Rise of the Bots

Information quality is further impaired by social bots, which can exploit all our cognitive loopholes. Bots are easy to create. Social media platforms provide so-called application programming interfaces that make it fairly trivial for a single actor to set up and control thousands of bots. But amplifying a message, even with just a few early upvotes by bots on social media platforms such as Reddit, can have a huge impact on the subsequent popularity of a post.

At OSoMe, we have developed machine-learning algorithms to detect social bots. One of these, Botometer, is a public tool that extracts 1,200 features from a given Twitter account to characterize its profile, friends, social network structure, temporal activity patterns, language and other features. The program compares these characteristics with those of tens of thousands of previously identified bots to give the Twitter account a score for its likely use of automation.

In 2017 we estimated that up to 15 percent of active Twitter accounts were bots—and that they had played a key role in the spread of misinformation during the 2016 U.S. election period. Within seconds of a fake news article being posted—such as one claiming the Clinton campaign was involved in occult rituals—it would be tweeted by many bots, and humans, beguiled by the apparent popularity of the content, would retweet it.

Bots also influence us by pretending to represent people from our in-group. A bot only has to follow, like and retweet someone in an online community to quickly infiltrate it. OSoMe researcher Xiaodan Lou developed another model in which some of the agents are bots that infiltrate a social network and share deceptively engaging low-quality content—think of clickbait. One parameter in the model describes the probability that an authentic agent will follow bots—which, for the purposes of this model, we define as agents that generate memes of zero quality and retweet only one another. Our simulations show that these bots can effectively suppress the entire ecosystem's information quality by infiltrating only a small fraction of the network. Bots can also accelerate the formation of echo chambers by suggesting other inauthentic accounts to be followed, a technique known as creating “follow trains.”

Some manipulators play both sides of a divide through separate fake news sites and bots, driving political polarization or monetization by ads. At OSoMe, we recently uncovered a network of inauthentic accounts on Twitter that were all coordinated by the same entity. Some pretended to be pro-Trump supporters of the Make America Great Again campaign, whereas others posed as Trump “resisters”; all asked for political donations. Such operations amplify content that preys on confirmation biases and accelerate the formation of polarized echo chambers.

Curbing Online Manipulation

Understanding our cognitive biases and how algorithms and bots exploit them allows us to better guard against manipulation. OSoMe has produced a number of tools to help people understand their own vulnerabilities, as well as the weaknesses of social media platforms. One is a mobile app called Fakey that helps users learn how to spot misinformation. The game simulates a social media news feed, showing actual articles from low- and high-credibility sources. Users must decide what they can or should not share and what to fact-check. Analysis of data from Fakey confirms the prevalence of online social herding: users are more likely to share low-credibility articles when they believe that many other people have shared them.

Another program available to the public, called Hoaxy, shows how any extant meme spreads through Twitter. In this visualization, nodes represent actual Twitter accounts, and links depict how retweets, quotes, mentions and replies propagate the meme from account to account. Each node has a color representing its score from Botometer, which allows users to see the scale at which bots amplify misinformation. These tools have been used by investigative journalists to uncover the roots of misinformation campaigns, such as one pushing the “pizzagate” conspiracy in the U.S. They also helped to detect bot-driven voter-suppression efforts during the 2018 U.S. midterm election. Manipulation is getting harder to spot, however, as machine-learning algorithms become better at emulating human behavior.

Apart from spreading fake news, misinformation campaigns can also divert attention from other, more serious problems. To combat such manipulation, we have recently developed a software tool called BotSlayer. It extracts hashtags, links, accounts and other features that co-occur in tweets about topics a user wishes to study. For each entity, BotSlayer tracks the tweets, the accounts posting them and their bot scores to flag entities that are trending and probably being amplified by bots or coordinated accounts. The goal is to enable reporters, civil-society organizations and political candidates to spot and track inauthentic influence campaigns in real time.

These programmatic tools are important aids, but institutional changes are also necessary to curb the proliferation of fake news. Education can help, although it is unlikely to encompass all the topics on which people are misled. Some governments and social media platforms are also trying to clamp down on online manipulation and fake news. But who decides what is fake or manipulative and what is not? Information can come with warning labels such as the ones Facebook and Twitter have started providing, but can the people who apply those labels be trusted? The risk that such measures could deliberately or inadvertently suppress free speech, which is vital for robust democracies, is real. The dominance of social media platforms with global reach and close ties with governments further complicates the possibilities.

One of the best ideas may be to make it more difficult to create and share low-quality information. This could involve adding friction by forcing people to pay to share or receive information. Payment could be in the form of time, mental work such as puzzles, or microscopic fees for subscriptions or usage. Automated posting should be treated like advertising. Some platforms are already using friction in the form of CAPTCHAs and phone confirmation to access accounts. Twitter has placed limits on automated posting. These efforts could be expanded to gradually shift online sharing incentives toward information that is valuable to consumers.

Free communication is not free. By decreasing the cost of information, we have decreased its value and invited its adulteration. To restore the health of our information ecosystem, we must understand the vulnerabilities of our overwhelmed minds and how the economics of information can be leveraged to protect us from being misled.

Authors: Filippo Menczer

Source: Scientific American

How Artificial Intelligence could drive the Electric Vehicles market

Artificial intelligence (AI) is rapidly evolving and becoming ubiquitous across virtually every industry. AI solutions allow organizations to achieve operational efficiencies, gain insights into customer behavior, measure key performance indicators (KPIs), and leverage the power of big data, among other things.

Similarly, the electric vehicles (EV) market has gained traction in recent years. It’s more common to see drivers cruising in EVs, whether a Tesla, Chevy Bolt, or Nissan Leaf. EVs are becoming popular among eco-conscious consumers because they offer more eco-friendly benefits than traditional gas-powered vehicles.

EVs have shown growth throughout the decade and great promise, but adoption rates have lagged in the U.S. compared to other countries.

Is it possible for AI to play a role in helping EV adoption in the U.S. and other countries? Here’s how the EV market could leverage AI to increase sales and create a more sustainable transportation system.

Looking at Current EV Adoption

The U.S. has noted increases in EV adoption rates, but rates are still on the low side compared to other regions of the world. According to data from the World Economic Forum, Norway, Iceland and Sweden lead the world in EV adoption.

One main reason other countries have adopted EVs on a larger scale is that it’s common for their governments to offer incentives to consumers. Various policies have incentivized EV purchases in Norway, but the World Economic Forum suggests this may not fly in other countries.

According to the Argonne National Laboratory, a U.S. Department of Energy (DOE) research center, nearly 2.4 million battery EVs have been sold since 2010. A critical aspect of the EV market is implementing the infrastructure to support charging. Many consumers may be hesitant to purchase or lease EVs because they worry about finding charging stations in their area.

The U.S. currently has almost 113,600 EV charging stations, with most of them located in California. The Biden administration announced a plan to allocate $5 billion in the next five years to build up the EV charging network, which will certainly aid in adoption rates.

How AI Can Speed up EV Adoption

Aside from government funding for infrastructure improvements, other factors will play a role in aiding EV adoption. An article from Forbes cites five major factors driving adoption, including:

• Emissions regulations
• Technology
• Cost
• Overcoming myths about the environmental impact of EVs
• A fast-changing EV market with various players (Volkswagen, Tesla, Hyundai, Kia, etc.)

AI can be used for various applications, so it’s worth exploring how it can be leveraged in the EV market to drive adoption.

Improving EV Batteries

One piece of technology necessary for EV development is electric batteries. Developing a suitable battery for an EV requires testing various material combinations, and that’s a time-consuming process. 

EV battery manufacturers can leverage AI solutions to sift through vast amounts of data much quicker than a human researcher. For example, a recent IBM project involved developing a battery capable of faster charging without nickel or cobalt. Researchers had to evaluate a set of 20,000 compounds to determine the battery’s electrolytes. Normally, it would take five years to process this data, but it only took nine days with the help of AI.

Additionally, AI can aid in testing batteries for EVs. Algorithms can be trained to predict how they will perform using only a small amount of data. Speeding up battery research and development will improve EVs, thus speeding up adoption.

Smoothing Out EV Charging Demand

A new project in Canada may help manage EV charging when demand is high. It was recently announced that the Independent Electricity System Operator (IESO) and the Ontario Energy Board (OEB) would support an AI project to improve EV charging management. BluWave-ai and Hydro Ottawa are leading the project, and it’s expected to enhance charging operations when energy is in peak demand.

The pilot project is called EV Everywhere. It uses AI to create an online service for drivers and pools batteries’ storage and charging capabilities. The system will automatically gauge customer interests and impacts, smooth out demand peaks, and allow people to capitalize on lower-cost charging during off-peak times.

Enhancing HMIs for Safety

Another factor driving the adoption of EVs is to ensure safety for drivers and passengers. One essential feature in an EV and most modern vehicles is the human-machine interface (HMI), which is needed for controlling and providing signals to various types of automated equipment, including the LED screens found in many EVs.

HMI systems that leverage AI solutions allow drivers to access a voice-enabled smart assistant, additional controls, better EV monitoring, and infotainment. AI-powered HMI systems will become more widely used, helping drive adoption.

AI has many use cases, especially in the EV market. It’ll be interesting to see how manufacturers and other companies leverage AI to encourage adoption.

Expect More AI Use Cases to Drive EV Adoption

It is becoming more popular for drivers to consider purchasing EVs, but adoption rates need to increase to create a more sustainable transportation system. AI will play a significant role in driving sales if more companies find innovative ways to use these solutions. It’s only a matter of time until EVs become the dominant mode of transportation, but leveraging AI will be critical in reaching that point.

Author: April Miller

Source: Open Data Science

How artificial intelligence will shape the future of business

From the boardroom at the office to your living room at home, artificial intelligence (AI) is nearly everywhere nowadays. Tipped as the most disruptive technology of all time, it has already transformed industries across the globe. And companies are racing to understand how to integrate it into their own business processes.

AI is not a new concept. The technology has been with us for a long time, but in the past, there were too many barriers to its use and applicability in our everyday lives. Now improvements in computing power and storage, increased data volumes and more advanced algorithms mean that AI is going mainstream. Businesses are harnessing its power to reinvent themselves and stay relevant in the digital age.

The technology makes it possible for machines to learn from experience, adjust to new inputs and perform human-like tasks. It does this by processing large amounts of data and recognising patterns. AI analyses much more data than humans at a much deeper level, and faster.

Most organisations can’t cope with the data they already have, let alone the data that is around the corner. So there’s a huge opportunity for organisations to use AI to turn all that data into knowledge to make faster and more accurate decisions.

Customer experience

Customer experience is becoming the new competitive battleground for all organisations. Over the next decade, businesses that dominate in this area will be the ones that survive and thrive. Analysing and interpreting the mountains of customer data within the organisation in real time and turning it into valuable insights and actions will be crucial.

Today most organisations are using data only to report on what their customers did in the past. SAS research reveals that 93% of businesses currently cannot use analytics to predict individual customer needs.

Over the next decade, we will see more organisations using machine learning to predict future customer behaviours and needs. Just as an AI machine can teach itself chess, organizations can use their existing massive volumes of customer data to teach AI what the next-best action for an individual customer should be. This could include what product to recommend next or which marketing activity is most likely to result in a positive response.

Automating decisions

In addition to improving insights and making accurate predictions, AI offers the potential to go one step further and automate business decision making entirely.

Front-line workers or dependent applications make thousands of operational decisions every day that AI can make faster, more accurately and more consistently. Ultimately this automation means improving KPIs for customer satisfaction, revenue growth, return on assets, production uptime, operational costs, meeting targets and more.

Take Shop Direct for example, which owns the Littlewoods and Very brands. This approach saw Shop Direct’s profits surge by 40%, driven by a 15.9% increase in sales from Very.co.uk. It uses AI from SAS to analyse customer data in real time and automate decisions to drive groundbreaking personalisation at an individual customer level.

AI is here. It’s already being adopted faster than the arrival of the internet. And it’s delivering business results across almost every industry today. In the next decade, every successful company will have AI. And the effects on skills, culture and structure will deliver superior customer experiences.

Author: Tiffany Carpenter

Source: SAS

How automated data analytics can improve performance

Data, data, data. Something very valuable to brands. They need it in order to make informed decisions and in the long term, make their brand grow. That part is probably common knowledge, right? What you are probably wondering is how big brands are choosing and using the right data analytics that will bring results. Find out the answer to that question here.

Data analytics to learn more about brand performance

Data analytics for risk management

Following on from the last point of big brands needing precise data to make informed decisions, they also need such data for risk management. Being able to grow as a brand is not just about knowing who their customers are, their intention to buy their product, etc., it is also about being able to foresee any potential risks and knocking them out of the park before they can cause any damage. Take for instance UOB bank in Singapore, who have devised a risk management system based on big data.

Data analytics to predict consumer behavior

As much as big brands need to look into the future, they also need to look to the past. Historical data can do wonders for future growth. Data analytics can be used to pinpoint patterns in consumer behavior. Using the data, they can potentially predict when a certain market may take a nosedive, as well as markets on an upward trend that are worth investing money into right now.

Data analytics for better marketing

A combination of data analytics looking at the past, present, and future of a big brand can make for better marketing, and in turn, more profit. By using data analytics to identify consumer needs and purchasing patterns, big brands can target with more personalized marketing, refine the overall consumer experience, and develop better products. Pay attention in your everyday life and you can already see examples of such data being used to market a product at you. A product you Googled once now appearing in your Facebook feed? Retargeting. Emails sounding like they are speaking directly to your needs? That’s because they are, since there are more than a few email marketing approaches. Data analytics was used to figure out exactly what you need.

There is one important trend occurring across the different ways that big brands are using data analytics to bring results. They all aim to understand consumers, in particular, the brands’ target audience. Whether that be what consumers think of their brand now, how they reacted toward them in the past, and how brands think consumers will act in the future because of detected patterns.

So, how are big brands using data analytics that will bring results? They are using them in a way that will help them better understand the consumer. 

Author: Steve Habazin

Source: Insidebigdata

How autonomous vehicles are driven by data

Understanding how to capture, process, activate, and store the staggering amount of data each vehicle is generating is central to realizing the future of  autonomous vehicles (AVs).

Autonomous vehicles have long been spoken about as one of the next major transformations for humanity. And AVs are already a reality in delivery, freight services, and shipping, but the day when a car is driving along the leafy suburbs with no one behind the wheel, or level five autonomy as it’s also known, is still far off in the future.

While we are a long way off from having AVs on our roads, IHS Markit reported last year that there will be more than 33 million autonomous vehicles sold globally in 2040. So, the revolution is coming. And it’s time to be prepared.

Putting some data in the tank

As with so many technological advancements today, data is critical to making AVs move intelligently. Automakers, from incumbents to Silicon Valley startups, are running tests and racking up thousands of miles in a race to be the leader in this field. Combining a variety of sensors to recognize their surroundings, each autonomous vehicle uses radar, lidar, sonar and GPS, to name just a few technologies, to navigate the streets and process what is around them to drive safely and efficiently. As a result, every vehicle is generating a staggering amount of data.

According toa report by Accenture, AVs today generate between 4 and 6 terabytes (TBs) of data per day, with some producing as much as 8 to 10 TBs depending on the number of mounted devices on the vehicle. The report says that on the low end, that means the data generated from one test car in one day is roughly the equivalent to that of nearly 6,200 internet users.

While it can seem a little overwhelming, this data contains valuable insights and ultimately holds the key in getting AVs on the road. This data provides insights into how an AV identifies navigation paths, avoids obstacles, and distinguishes between a human crossing the road or a trash can that has fallen over in the wind. In order to take advantage of what this data can teach us though, it must be collected, downloaded, stored, and activated to enhance the decision-making capabilities of each vehicle. By properly storing and managing this data, you are providing the foundation for progress to be made securely and speedily.

Out of the car, into the ecosystem

The biggest challenge facing AV manufacturers right now is testing. Getting miles on the clock and learning faster than competitors to eliminate errors, reach deadlines, and get one step closer to hitting the road. Stepping outside of the car, there is a plethora of other elements to be considered from a data perspective that are critical to enabling AVs.

Not only does data need to be stored and processed in the vehicle, but also elsewhere on the edge and some of it at least, in the data center. Test miles are one thing, but once AVs hit the road for real, they will need to interact in real-time with the streets they are driving on. Hypothetically speaking, you might imagine that one day gas stations will be replaced by mini data centers on the edge, ensuring the AVs can engage with their surroundings and carry out the processing required to drive efficiently.

Making the roads safer

While it might seem that AVs are merely another technology humans want to use to make their lives easier, it’s worth remembering some of the bigger benefits. The U.S. National Highway Traffic Safety Administration has stated that with human error being the major factor in 94% of all fatal accidents, AVs have the potential to significantly reduce highway fatalities by addressing the root cause of crashes.

That’s not to say humans won’t be behind the wheel at all in 20 years, but as artificial intelligence (AI) and deep learning (DL) have done in other sectors, they will augment our driving experience and look to put a serious dent in the number of fatal road accidents every year, which currently stands at nearly 1.3 million.

Companies in the AV field understand the potential that AI and DL technology represents. Waymo, for example, shared one of its datasets in August 2019 with the broader research community to enable innovation. With data containing test miles in a wide variety of environments, from day and night, to sunshine and rain, data like this can play a pivotal role in preparing cars for all conditions and maintaining safety as the No. 1 priority.

Laying the road ahead

Any company manufacturing AVs or playing a significant role in the ecosystem, from edge to core, needs to understand the data requirements and implement a solid data strategy. By getting the right infrastructure in place ahead of time, AVs truly can become a reality and bring with them all the anticipated benefits, from efficiency of travel to the safety of pedestrians.

Most of the hardware needed is already there: radars, cameras, lidar, chips and, of course, storage. But understanding how to capture, process, activate, and store the data created is central to realizing the future of AVs. Data is the gas in the proverbial tank, and by managing this abundant resource properly, you might just see that fully automated car in your neighborhood sooner than expected.

Author: Jeff Fochtman

Source: Informationweek

How Big Data leaves its mark on the banking industry

Did you know that big data can impact your bank account, and in more ways than one? Here's what to know about the role big data is playing in finance and within your local bank.

Nowadays, terms like ‘Data Analytics,’ ‘Data Visualization,’ and ‘Big Data’ have become quite popular. These terms are fundamentally tied predominantly to matters involving digital transformation as well as growth in companies. In this modern age, each business entity is driven by data. Data analytics are now very crucial whenever there is a decision-making process involved.

Through this tool, gaining better insight has become much easier now. It doesn’t matter whether the decision being considered has huge or minimal impact; businesses have to ensure they can access the right data to move forward. Typically, this approach is essential, especially for the banking and finance sector in today’s world.

The role of Big Data

Financial institutions such as banks have to adhere to such a practice, especially when laying the foundation for back-test trading strategies. They have to utilize Big Data to its full potential to stay in line with their specific security protocols and requirements. Banking institutions actively use the data within their reach in a bid to keep their customers happy. By doing so, these institutions can limit fraud cases and prevent any complications in the future.

Some prominent banking institutions have gone the extra mile and introduced software to analyze every document while recording any crucial information that these documents may carry. Right now, Big Data tools are continuously being incorporated in the finance and banking sector. 

Through this development, numerous significant strides are being made, especially in the realm of banking. Big Data is taking a crucial role, especially in streamlining financial services everywhere in the world today. The value that Big Data brings with it is unrivaled, and, in this article, we will see how this brings forth positive results in the banking and finance world.

The underlying concept 

A 2013 survey conducted by the IBM’s Institute of Business Value and the University of Oxford showed that 71% of the financial service firms had already adopted analytics and big data. Financial and banking industries worldwide are now exploring new and intriguing techniques through which they can smoothly incorporate big data analytics in their systems for optimal results.

Big data has numerous perks relating to the financial and banking industries. With the ever-changing nature of digital tech, information has become crucial, and these sectors are working diligently to take up and adjust to this transformation. There is significant competition in the industry, and emerging tactics and strategies must be accepted to survive the market competition. Using big data, firms can boost the quality and standards of their services.

Perks associated with Big Data

Analytics and big data play a critical role when it comes to the financial industry. Firms are currently developing efficient strategies that can woo and retain clients. Financial and banking corporations are learning how to balance Big Data with their services to boost profits and sales. Banks have improved their current data trends and automated routine tasks. Here are a few of the advantages of Big Data in the banking and financial industry:

Improvement in risk management operations

Big Data can efficiently enhance the ways firms utilize predictive models in the risk management discipline. It improves the response timeline in the system and consequently boosts efficiency. Big Data provides financial and banking organizations with better risk coverage. Thanks to automation, the process has become more efficient.Through Big Data, groups concerned with risk management offer accurate intelligence insights linked to risk management.

Engaging the workforce

Among the most significant perks of Big Data in banking firms is worker engagement. The working experience in the organization is considerably better. Nonetheless, companies and banks that handle financial services need to realize that Big Data must be appropriately implemented. It can come in handy when tracking, analyzing, and sharing metrics connected with employee performance. Big Data aids financial and banking service firms in identifying the top performers in the corporation.

Client data accessibility

Companies can find out more regarding their clients through Big Data. Excellent customer service implies outstanding employee performance. Aside from designing numerous tech solutions, data professionals will assist the firm set performance indicators in a project. It will aid in injective analytic expertise in multiple organizational areas. Whenever there is a better process, the work processes are streamlined. The banking and financial firms can leverage improved insights and knowledge of customer service and operational needs.

Author: Matt Bertram

Source: Smart Data Collective

How data can aid young homeless people

What comes to mind when you think of a “homeless person”? Chances are, you’llpicture an adult, probably male, dirty, likely with some health conditions, including a mental illness. Few of us would immediately recall homeless individuals as family members, neighbors, co-workers and other loved ones. Fewer still arelikely aware of how many youths (both minors and young adults) experience homelessness annually.

Homeless youth is a population who can become invisible to us in many ways. These youth may still be in school, may go to work, and interact with many of our public and private systems, yet not have a reliable and safe place to sleep, eat, do homework and even build relationships.

Youth experiencing homelessness is, in fact, far more prevalent than many people realize, as the Voices of Youth Count research briefs have illustrated. Approximately 1 in 10 young adults (18-25) and 1 in 30 youth (13-17) experience homelessness over the course of a year. That’s over 4 million individuals.

When I worked for the San Bernardino County Department of Behavioral Health, we ran programs specifically targeting homeless youth. The stories of lives changed from supportive care is still motivating!Myrole at the County focused primarily on data. At SAS, I have continued to explore ways data can support whole person care, which includes the effects of homelessness on health. 

I see three primary ways data can be powerful in helping homeless youth: 

1. Data raises awareness

Without good data, it’s hard to make interventionithout good data, it’sWithout good data, it’s hard to make interventions. Health inequities is a good example of this: If we don’t know where the problem is, we can’t change our policies and programs.

The National Conference of State Legislatures has compiled a range of data points about youth homelessness in the United States and informationon related policy efforts. This is wonderful information, and I appreciate how they connect basic data with policy.

At the same time, this kind of data can be complicated to compile. Information about youth experiencing homelessness can be siloed, which inhibits a larger perspective, like a regional, statewide, or even national view. We also know there are many intersections with other public and private systems, including education, foster care, criminal justice, social services, workforce support and healthcare. Each system has a distinct perspective and data point.

What would happen if we were able to have a continuous whole person perspective of youth experiencing homelessness? How might that affect public awareness and, by extension, public policy to help homeless youth?

2. Data informs context and strengths

While chronic health conditions are often present with homeless youth, this is also an issue with family members, leading to family homelessness. First off, this is an important example of not looking at people at just individuals, but as part of a bigger system. That fundamentally requires a more integrative perspective.

Further, homeless youth experience higher rates of other social factors, such as interactions with foster care, criminal justice, and educational discipline (e.g., suspensions). Add on top of that other socio-economic contexts, including racial disparities and more youth from the LGBTQ+ communities.

Just as I talked about the evaluation of suffering in general, having a more whole person perspective on homelessness is critical in understanding the true context of what may be contributing to homelessness… as well as what will help with it.

It is easy to focus on all the negative outcomes and risk factors of homelessness in youth. What happens when we can start seeing folks experiencing homelessness as loved and meaningful members of our communities? Data that provides more holistic perspectives, including strengths, could help shift that narratives and even combat stigma and discrimination.

In my role at San Bernardino County, I helped oversee and design program evaluation, including using tools, like the Child and Adolescent Needs and Strengths (CANS), to assess more holistic impacts of acute programs serving homeless youth. Broadening out our assessment beyond basic negative outcomes to includemetrics like resilience, optimism, and social support not only reinforces good interventions, but also helps us to see the youth experiencing homelessness as youth worthy of investment.

That’s invaluable.

3. Data empowers prevention and early intervention 

Finally, homelessness is rarely a sudden event. In most cases, youth and their families experiencing homelessness have encountered one or more of our community systems before becoming homeless. I’ve talked before about using more whole person data to proactively identify people high-risk people across public (especially health) systems.

This approach can lead to early identification of people at risk of homelessness. If we can identify youth and family in an early encounter with health, social services, foster care or even the criminal justice system, could we better prevent homelessness in the first place? Some people will still experience homelessness, but could this same approach also help us better identify what kinds of interventions could reduce the duration of homelessness and prevent it from recurring?

With whole person data, we can continue to refine our interventions and raise more awareness of what best helps youth experiencing homelessness. For instance, research has recognized the value of trauma-informed care with this population. The National Child Traumatic Stress Network has a variety of information that can empower anyone to better help homelessyouth.

In honor of National Homeless Youth Awareness Month and recognizing the importance of homelessness in general, I encourage you to explore some of these resources and read at least one to become more aware of the reality of the experience of homeless youth. That’s the first step in moving us forward.

Author: Josh Morgan

Source: SAS

How Data Science is Changing the Entertainment Industry

Beyond how much and when, to what we think and how we feel

Like countless other industries, the entertainment industry is being transformed by data. There’s no doubt data has always played a role in guiding show-biz decision-making, for example, in the form of movie tracking surveys and Nielsen data. But with the ever-rising prominence of streaming and the seamless consumption measurement it enables, data has never been more central to understanding, predicting, and influencing TV and movie consumption.

With experience as both a data scientist in the entertainment space and a researcher of media preferences, I’ve had the fortune of being in the trenches of industry analyzing TV/movie consumption data and being able to keep up with media preferences research from institutions around the world. As made evident by the various citations to come, the component concepts presented here themselves aren’t anything new, but I wanted to apply my background to bring together these ideas in laying out a structured roadmap for what I believe to be the next frontiers in enhancing our ability to understand, predict, and influence video content consumption around the world. While data can play a role at many earlier phases of the content lifecycle — e.g. in greenlighting processes or production — and what I am about to say can be relevant in various phases, I write mainly from a more downstream perspective, nearer to and after release as content is consumed, as cultivated during my industry and academic work.

Beyond Viewing and Metadata

When you work in the entertainment space, you end up working a lot with title consumption data and metadata. To a large extent, this is unavoidable — all “metadata” and “viewing data” really mean is data on what’s being watched and how much — but it’s hard to not start sensing that models based on such data, as commonly seen in content similarity analyses, output results that fall into familiar patterns. For example, these days when I see “similar shows/movies” recommendations, a voice in my head goes, “That’s probably a metadata-based recommendation ,” or, “Those look like viewership-based recommendations,” based on what I’ve seen during my work with such models. I can’t be 100% sure, of course, and the voice is more confident with smaller services that likely use more off-the-shelf approaches; on larger platforms, recommendations are often seamless enough that I’m not thinking about flaws, but who knows what magic sauce is going into them?

I’m not saying viewing data and metadata will ever stop being important, nor do models using such data fail to explain ample variance in consumption. What I am saying is that there is a limit to how far solely these elements will get us when it comes to best analyzing and predicting viewership— we need new ways to enhance understanding of viewers and their relationship with content. We want to understand and foresee title X’s popularity at time point A beyond, “It was popular at A-1, it will be popular at A,” or, “title Y, which is similar to X, was popular, so X will be popular”, especially since often, data at A-1 or on similarity between X and Y may not be available. Let’s talk about one type of data that I think will prove critical in enhancing understanding of and predictive capacity concerning viewership moving forward.

Psychometrics: Who is Watching and Why

People love to talk demographics when it comes to media consumption. Indeed, anyone who’s taken a movie business class is likely to be familiar with the “four quadrant movie”, or a movie that can appeal to men and women over and under the ages of 25. But demographics are limited in their explanatory and predictive utility in that they generally go as far as telling us the who but not necessarily the why.

That’s where psychometrics (a.k.a. psychographics) can provide a boost. Individuals in the same demographic can easily have different tendencies, values, preferences; an example would be the ability to divide men or women into people who tend to be DIYers, early adopters, environmentalists, etc. based on their measured characteristics across various dimensions. Similarly, people of different demographics can easily have similar characteristics, such as being high in thrill-seeking, being open to new experiences, or identifying as left/right politically. Such psychometric variables have indeed been shown to influence media preference — for example, agreeable people like talk shows and soaps more, higher sensation seeking individuals like violent content more — and improve the capacity of recommendation models. My own research has shown that even abbreviated psychometric measures can produce an improvement in model fit to genre preference data compared to demographic data alone. Consumer data companies have already begun to recognize the importance of psychometric data, with many of them incorporating them in some form into their services.

Psychometric data can be useful at the individual-level at which they are often collected, or aggregated to provide group-level — audience, userbase, country, so on — psychometric features of various kinds. Some such data might come ‘pre-aggregated’ at the source, as is the case with, for example, Hofstede’s cultural dimensions. In terms of collection, when direct collection for all viewers in an audience isn’t feasible (e.g. when you can’t survey millions of users), a “seed” set of self-report survey data from responding viewers could be used to impute the values to similar non-respondents using nearest neighbor methods. Psychometric data can also be beneficial in cold-start problem scenarios — if you don’t have direct data about what a particular audience watches or how much they would watch particular titles, wouldn’t data about their characteristics that point to the types of content are likely to want be useful?

Consumption as Viewer-Content Trait Interaction

The above section discusses psychometrics in particular, but zooming out a bit, what it is more broadly pushing for is an expansion of the viewer/audience feature space beyond the demographic and behavioral. This is because all consumption is inherently an interaction between the traits of a viewer and the traits of a piece of content. This concept is simpler and more well-tread than it may sound; all it really means is that some element of a viewer (viewer trait) means they are more (or less) drawn to some element of a piece of content (content trait). Even familiar stereotypes about genre preferences — children are more into animation, men are more into action, etc. — inherently concern viewer-content trait interactions (viewer age-content genre, viewer sex-content genre in above examples), and the aforementioned research on viewer psychometrics effects on content preferences also fall under this paradigm.

The larger the array of viewer traits we have, the more things we can consider might interact with some kind of content trait to impact their interest in consuming the title. Conversely, this also means that it is beneficial to have new forms of data title-side as well. It can seem like people are more readily ‘get deep’ with title-side data, in the form of metadata (genre, cast, crew, studio, awards, average reviews, etc.), than they do with viewer-side data, but there’s still room for expansion title-side, especially if one is expanding viewer-side data as suggested above through collection of psychometrics and the like. Tags and tagging are a good place to start in this regard. Human tagging can particularly be beneficial by capturing latent information still difficult for machines to detect on their own — e.g. humor, irony, sarcasm, etc. — but automated processes can provide useful baseline content tags of a consistent nature. However, these days, tags are just the start when it comes to generating additional title-side data. It’s possible to engineer all sorts of features from the audio and video of titles, as well as to extract the emotional arc of a story from text.

Once you consider consumption from the viewer-content interaction lens and expand data collection on both the viewer and title sides, the possibilities really open up. You could, for example, code the race/ethnicity and gender of characters in a title and see how demographic similarity between the title cast/crew and the typical users of a streaming platform can impact the title’s success. Or maybe you want to code titles for their message sensation value to see how that’s associated with the title’s appeal to a particular high sensation-seeking group. Or perhaps you want to use data from OpenSubtitles or the like to determine the narrative arc type of all the titles in your system and see if any patterns arise as to the appeal of certain arcs to individuals of certain psychographics.

Parsing the Pipeline: Perception, Interest, Response

Lastly, there needs to be a more granular consideration of the consumption pipeline, from interest to response. Though easily lumped together as “good” signals of a consumer’s feelings about a title, being interested in, watching, and liking a piece of content are entirely different things. Here’s how the full viewing process should be parsed out when possible, separated broadly into pre-consumption and post-consumption phases.

Perception (Pre-consumption): Individuals of different demographics, and presumably of different psychographics, can perceive the same media product differently. These perceptions can be shaped by the elements of a product’s brand design, font, colors, and advertisements. Perception arguably has important effects on the next phase in the pipeline.

Interest, and Selection (Pre-consumption): First off, though related and the former certainly increases the likelihood of the latter, it is important to note that interest (a.k.a. preference) is not the same has selection (a.k.a. choice). Though analyses regarding one may often be relevant to the other, we cannot always assume that an individual who expresses interest in something or has a high likelihood of being interested in something will always choose to consume it. This is well exemplified by models like the Reasoned Action Model, within which framework an individual who feels favorably about watching a movie may not watching it to perceived unfavorable norms about watching said movie. Examining factors driving interest-selection conversion may be beneficial.

Response (Post-consumption): Lastly, there is how individuals feel after watching a piece of content. This could be as simple as whether they liked it or not; and though it can be tempting to equate high viewership with wow, people really like that movie when looking at a dataset, it’s critical to remember that how much people watch something and whether they like it are related but ultimately separate things, as anyone who was stoked for a movie then crushed by its mediocrity can attest; my own research has shown that the effects at play with interest in unseen content can differ from, even be the opposite of, the effects at play with liking of seen content. Beyond liking, responses can also include elements such as how viewers felt about the content emotionally, how much they related with the characters, to what degree they were immersed into the storyline, and more.

Media preference and consumption does not need to be considered a singular, stationary process, but instead, separated out this way, a fluid modular, process where strategic management of upstream processes can impact the likelihood of desired outcomes, whatever they may be, down the line. How can we selectively optimize perception of a media product across different demographic and psychographic groups to get maximum interest in a title — or perhaps, optimize the desired downstream outcome? How can we optimally convert interest into selection? Can certain upstream perceptions or overly high levels of interest interact adversely with the content of a certain title such that the ultimate response to the title is more negative than it would have been had perceptions been different or interest less extreme? In addition, though I provide potential key mechanisms of relevance to each step of the pipeline, certain mechanisms may be of relevance at multiple phases or across different phases of the pipeline — for example, (potential) viewer-character similarity may impact perception of and interest in a title after exposure to advertising, while social network effects may mean the post-consumption responses of certain individuals heavily influence pre-consumption interest among other individuals.

Conclusion

As an industry, we’ve only begun to scratch the surface of how data can help us understand, predict, and influence content consumption, and these are just some of my thoughts on what I believe will be important considerations as data science becomes ever more prevalent and critical in the entertainment space. Audience psychometrics will help enhance understanding of audiences beyond what demographics can do alone; considering interactions between new audience and content features will provide superior strategic insights and predictive capacity; and a nuanced consideration of the full consumption pipeline from interest to response will help optimize desired outcomes.

Author: Danny Kim

Source: Towards Data Science

How do automation and robotics differ?

We live in a tech-driven world, but with all the technologies and innovative solutions at our disposal, it can be difficult to figure out what tech solutions are right for your business. Oftentimes, business leaders who are looking into tech innovation and new solutions for their companies are overwhelmed by the sheer number of possibilities.

That said, there always seems to be some confusion as to the difference between automation and robotics, their unique and combined uses, and more. Could both help your company thrive and improve your production and operational processes? Absolutely, but you need to differentiate between the two first.

Here are the key differences between automation and robotics and how to use them to your advantage.

What is automation?

Automation is the use of automatic hardware or software to automate specific tasks or a number of processes in a business. You can implement automation in a variety of ways, such as a custom solution for your company, as a part of a robotics solution in your facility, or as a SaaS solution.

The applications for automation are virtually endless nowadays, as there is a piece of automation equipment or a piece of software for almost any use and any department in an organization. For example, according to the latest Ecommerce trends, automation has become a big part in operations management, but also marketing, customer service, and more.

But Ecommerce is just one of many industries where automation has countless uses, with the goals of automating repetitive tasks, minimizing financial and time waste, and maximizing performance. As a software solution, you can use automation for:

• Repetitive and menial tasks
• Warehouse management
• Order processing from numerous sales channels
• Team monitoring, workflow, and communication
• Scheduling and posting on social media
• Buying advertising space across the web
• And much more

What is robotics?

On the other hand, robotics is an interdisciplinary branch of engineering and computer science, involving the design, construction, operation and use of robotics. As the name implies, a robotic solution is a piece of sophisticated equipment or machinery used to assist humans and businesses overall in their daily operations.

The key difference between automation and robotics is that the former is used to automate specific tasks while robots are machines that can be programmed to complete a variety of different tasks. With robotic process automation, which we’ll talk about in a minute, you can combine the two for maximum effect.

Robotic solutions are primarily used for industrial purposes, but numerous other industries are increasingly leveraging this technology to boost operational efficiency.

Ecommerce is increasingly using robotics to improve order processing, warehouse management and goods retrieval, as well as storing, packaging, and shipping. In manufacturing, engineering, and construction, robotic solutions help improve worksite safety, accuracy, and time to completion.

Understanding robotics process automation

Robotics process automation is the process of designing, building, and implementing software robots that mimic human actions and is often referred to as software robotics. Using artificial intelligence and machine learning, robotics process automation is seeing widespread implementation across the world.

Software robotics is a big part of  digital transformation and the implementation of smart solutions across the business sector. One of the most obvious use cases is the implementation of chatbot technology and conversational AI in general.

But RPA extends far beyond help desk support and chatbots, and reaches into complex applications in everything from manufacturing to finance, administration, risk mitigation, and more.

Using automation as a software solution

As mentioned before, automation comes in many forms, and oftentimes you will need to implement it as a SaaS solution or use automation features as a part of a dedicated tool. Automation can be found all around the web, driving marketing and sales processes, employee management and performance tracking, all the way to onboarding and more.

From an online video editor used to fuel your video marketing efforts, all the way to automated workflow and team management tools, you can use software automation to cut financial waste and improve efficiency. You can also build a standalone solution specifically for your business, leveraging AI and machine learning to design a powerful proprietary automation tool to fix a unique pain-point in your company.

Whether it’s a part of a web-based tool or a SaaS product, or if it’s a custom solution, software tools are nowadays commonly used to automate many menial and even complex tasks. Needless to say, this will help empower employees and boost satisfaction in the workplace.

Enhancing security with automation and robotics

Combining software automation and robotics is one of the best ways to enhance business cyber-attacks, while robotics can be used to elevate the physical security of your business.

Now that smart robots are increasingly substituting on-site security, business leaders can use robots to patrol and oversee the business premises with ease. Security robots can connect with other sensors and cameras to create a comprehensive security system.

Combined, software and robotic security allow managers and business owners to better protect customer data, their employees, as well as all sensitive business data in general.

Over to you

Automation and robotics are two very different tech sectors that, combined or used separately, can enhance business operations and overall efficiency. Now that you know what they are and understand some of the ways these technologies can help you, go ahead and streamline your tech investment strategy for 2022 and beyond.

Author: Nikola Sekulic

Source: Datafloq

How Greece set an example using online volunteering to battle COVID-19

Assistant Volunteer, a project of Nable Solutions, was born during the HackCoronaGreece online hackathon to better coordinate the efforts of volunteers. Today, Assistant Volunteer’s platform is part of the Greek Ministry of Health’s official response to eradicating the pandemic. 

This year, online hackathons have proven to be a great source of ideation for easily scalable solutions during crises. From a shortage of medical equipment to caring for patients remotely, solutions to better manage the COVID-19 outbreak flourished globally. However, it was still to be discovered whether these solutions could be developed into mature products able to be integrated into the official’s response programs.

On April 7th-13th, in Berlin and Athens, the global tech community tackled the most pressing problems Greece faced due to COVID-19 outbreak during the HackCoronaGreece online hackathon organized by Data Natives, eHealth Forum and GFOSS with the support of GreeceVsVirus (an initiative by the Greek Ministry of Digital Governance, Ministry of Health, Ministry of Research & Innovation). Just two months later, Assistant Volunteer, matured its solution to the final stages of development and was selected by the Greek Ministry of Health to officially contribute to managing the COVID-19 pandemic in Greece. 

The era of volunteering

COVID-19 paved the way to a new era of volunteerism in response to the crisis. Even though isolated from each other, volunteer movements across the globe found ways to dedicate their time and efforts to help the ones in need and introduce innovative and effective ways of helping humanity.

According to the United Nations, in Europe and Central Asia, the volunteer movement has been officially recognized by some governments for their services provided by volunteers during the COVID-19 pandemic. That’s exactly the case with HackCoronaGreece and the solutions that have been created by diverse communities.

One such solution, Assistant Volunteer, recognizes the problem of coordination – when thousands of people are gathering for a good cause, their efforts deserve outstanding management to maximize positive effects. 

What is Assistant Volunteer?

Assistant Volunteer was developed as part of the HackCoronaGreece hackathon by Nable Solutions, an award-winning startup providing software solutions with a social cause. Assistant Volunteer is an easy-to-use volunteer management software platform for organizations and government agencies. It can be configured to support organizations of all types and sizes to achieve modernization and upgrade of the operations, seamlessly with their workflow. Through the modular architecture design, organisations can coordinate volunteers through the web app and mobile app. 

Any organization can register, create a profile, come up with actions needed, engage with the database of volunteers, track performance & measure impact.

Assistant Volunteer competed with 14 other teams to be selected in the finale of the HackCoronaGreece hackathon and continue the development of their idea. The solution was recognized by the Greek Ministry of Health and selected for assistance in further development. 

Multinational pharma giant MSD Ssupports the project

Another influential supporter of the project is MSD, a pharmaceutical multinational company that contributed with an award for Assistant Volunteer wich is a monetary prize of 7.000 EUR. 

Previously, MSD Greece donated 100,000 euros to the Ministry of Health “to strengthen the national greek health system and to protect its citizens”. 

MSD also donated 800,000 masks to New York and New Jersey. Working with Bill and Melinda Gates Foundation and other healthcare companies, MSD contributes to pushing the development of the vaccine forward, diagnostic tools, and treatments to treat COVID-19 as soon as possible.

The Greek Ministry of Health included Assistant Volunteer in their official efforts to fight the pandemic and facilitated the population of the platform with 10000 volunteer profiles. Now, organizations can take the next steps in coordinating the volunteer movement in Greece and, potentially, beyond.

Author: Evgeniya Panova

Source: Dataconomy

How Machine Learning is Taking Over Wall Street

Well-funded financial institutions are in a perpetual tech arms race, so it’s no surprise that machine learning is shaking up the industry. Investment banking, hedge funds, and similar entities are employing the latest machine learning techniques to gain an edge on the competition, and on the markets. While the reality today is that machine learning is mostly employed in the back office–for tasks such as credit scoring, risk management, and fraud detection–this is about to change dramatically.

Machine learning is migrating to where the action is: financial market trading. Once leading-edge Wall Street platforms that companies invested many millions in are soon to become obsolete due to machine learning. Understanding how disrupting Wall Street will change and evolve and why it matters is key to navigating the opportunities ahead.

Algorithm Trading

Algorithmic trading now dominates the derivative, equity, and foreign exchange trading markets. These trading strategies can be complex, but the essentials are straightforward: program a set of rules that takes market data as input and apply basic models (10 -day moving average) to generate an automated trade workflow. Over the years, these strategies have moved beyond simple time-series momentum and mean revision models to more exotic name strategies like snipes, slicers, and boxers. Evolved over decades, algorithm trading has replaced much of the manual trade order flow with faster static rules-based strategies. What was once cutting edge is now an inherent disadvantage. Static rules, no matter how complex, may work well in relatively stable markets but can’t react to evolve rapidly changing market conditions.

A machine learning algorithm’s clear advantage is it learns from experience and is not static. Employing massive datasets and pattern recognition, these algorithms produce models that learn from experience and are orders of magnitude more powerful than old-school algorithmic trading models. Decisions on how and when to trade will be made in some cases by using multi-agent systems that can act autonomously. At some point, these static algorithms will be no match for more nimble machine learning algorithms.

Why it Matters: Reskill and Upskill

Companies that make use of algorithm trading need to reskill or risk getting left behind. In a winner-take-all market, companies employing only slightly more advanced techniques like machine learning will continuously win a bigger share of the market. In addition to machine learning, businesses should expect an increased demand for data engineers, data scientists, MLOps specials, and others that can handle this sophisticated workflow.

High-Frequency Trading Agents

High-frequency trading (HFT) is the flashy cousin of algorithmic trading. Employing similar rules-based models, or even predictive analytics, these strategies operate at a much more rapid pace,; completing hundreds of stock trades in nanoseconds versus longer time range algorithmic trading strategies. High-frequency trading also relies on massive hardware and bandwidth infrastructure investment that often requires system colocation next to major exchanges. Given its sophistication, only 2% of financial trading firms employ high-frequency trading, yet at its peak, it accounted for 10 to 43% of stock trading volume on any given day.

The ingredients for HFT–massive computing power, high frequency streaming big data, and ultrafast connections–are all areas where deep learning and machine learning workflows excel. 

Pre-trained models can prevent machine learning and deep learning algorithms from becoming speed-limiting factors. Coupled with techniques such as deep reinforcement learning, HTF is primed for another technological leap. However, given its increased complexity, it will remain the domain of a relatively few, but highly profitable, firms. 

Why it’s Important: Trouble Ahead

The Flash Crash that occurred on May 6, 2010 caused trillions of dollars of market equity to be wiped out in an instant (36 minutes to be precise). Regulators have struggled to keep up with algorithmic trading and high-frequency trading, and will doubtless be hard-pressed to stay ahead of the next generation. HTF AI agents will require much more sophisticated risk monitoring and compliance systems that in turn will need to employ machine learning to monitor.

Risk Assessment Platforms

Despite the vast sums invested in technology by financial institutions, the humble Excel spreadsheet remains the number one applicationon Wall Street. Risk departments, charged with ensuring traders don’t make calamitous errors, are no exception. Even the better-equipped firms employ software that relies on rule sets and analytics that are apt at catching known risks but are poorly equipped to identify evolving market risk.  

The nature of a robust risk assessment platform is a kind of catch-all. Risk scales from individual trades, to companies, industry, country, and global risk profiling. Risk can be quantifiable, but often a risk assessment may need to rely on alternative data. Machine learning’s adaptability and flexibility make it a natural successor to current risk assessment software. Both supervised and unsupervised machine learning techniques can be employed to layer on more sophisticated risk strategies. Anomaly detection used to identify outliers is one such technique that can be readily employed to identify the rare events that are characteristic of risk modeling.  

Why it Matters: Risk and Repeat

The recent implosion of Archegos Capital in March cost some of the world’s most sophisticated banks to lose up to $10 billion, highlighting the poor systems and oversight that many financial institutions face with have to trade risk exposure. Similar risk failure, albeit of a smaller magnitude, continues to abound despite the lesson learned and trillions lost due to the risk failure that gave rise to the 2007 financial crisis. Risk departments are finally waking up to the inherent advantages of pattern recognition machine learning versus manual and backward-looking analytics tools. Add to this the increased complexity due to, you guessed it, machine learning trading strategies.

OMS Trading Platforms

Retail traders have flocked to online trading platforms like Robinhood, Fidelity, and E*Trade. The institutional professionals use more advanced systems called OMS (order management systems) from companies like B2Broker, Charles River, Interactive Brokers, and others. These institutional trading platforms all execute the same basic workflow. Financial market data is fed in; a set of static trading, risk, and compliance rules are applied; buy and sell orders are generated; the order book is updated, and trade analytics reports are generated. 

Traditionally, these platforms were closed systems. Many provide limited APIs that allow customization of various aspects such as data feeds, order flow, and algorithms, but most work only within the confines of their particular platform. Advanced hedge fund traders are employing sophisticated machine learning and deep learning techniques that utilize platforms like Tensorflow, Keras, PyTorch, and similar frameworks and libraries. Deep learning techniques such as deep reinforcement learning, NLU (natural language understanding), and transfer learning require these platforms. These models often require alternative data whose unstructured format does not readily make itself suitable for the structured time-series format many of these present trading platforms require. 

Why it’s Important: From Closed to Open

At some point, this equation will flip. Trading platforms are very good at order workflow and trade analytics. However, data profiling, data transformation, and machine learning algorithms need something much more flexible, adaptive, and open. The existing dominant market players will need to adopt a more open API approach that gives full access to every stage of the order workflow. At some point over the next 5 years, this in turn will lead to adoption by retailed brokers and bring machine learning trading to the masses. 

From Leader to Laggard

For the last few decades, Wall Street has been a clear leader in rolling out complex platforms such as algorithm trading and high-frequency trading (HTF), and other innovative trading strategies.  However, many of these systems rely on static rules-based systems or predictive analytics at best. Other companies that fully embraced machine learning and deep learning earlier have come to dominate sectors of their industry. Expect a similar shakeout in financial institutions as some companies go all-in on artificial intelligence and become the next generation of technology leaders.  

Author: Sheamus McGovern

Source: Open Data Science

How patent data can provide intelligence for other markets

Patents are an interesting phenomenon. Believe it or not, the number one reason why patent systems exist is to promote innovation and the sharing of ideas. Simply put, a patent is really a trade. A government has the ability to give a limited monopoly to an inventor. In exchange for this exclusivity, the inventor provides a detailed description of their invention. The application for a patent needs to include enough detail about the technology that a peer in the field could pick it up and understand how to make or practice that invention. Next, this description gets published to the world so others can read and learn from it. That's the exchange. Disclose how your invention is made, in enough detail to replicate, and you can get a patent.

It gets really interesting when you consider that the patent carries additional metadata with it. This additional data is above and beyond the technical description of the invention. Included in this data are the inventor names, addresses, the companies they work for (the patent owner), the date of the patent filing, a list of related patents/applications, and more. This metadata and the technical description of the invention make up an amazing set of data identifying research and development activity across the world. Also, since patents are issued by governments, they are inherently geographic. This means that an inventor has to apply for a patent in every country where they want protection. Add the fact that patents are quite expensive, and we are left with a set of ideas that have at least passed some minimal value threshold. That willingness to spend money signals a value in the technology, specifically a value of that technology in the country/market where each patent is filed. In many ways, if you want to analyze a technology space, patent data can be better than analyzing products. The technology is described in substantial detail and, in many cases, identifies tech that has not hit the market yet.

Breaking down a patent dataset

Black Hills IP has amassed a dataset of over 100 million patent and patent application records from across the world. We not only use data published by patent offices, but we also run proprietary algorithms on that data to create additional patent records and metadata. This means we have billions of data points to use in analysis, and likely have the largest consolidated patent dataset in the world. In the artificial intelligence (AI) space alone, we have an identified set of between one hundred thousand and two hundred thousand patent records. This has been fertile ground for analysis and insight.

Breaking down this dataset, we can see ownership and trends around foundational and implementational technologies. For example, several of the large US players are covering their bases with patent filings in multiple jurisdictions, including China. Interestingly enough, the inverse is not necessarily shown. Many of the inventions from Chinese companies have their patent filings (and thus protection) limited to just China. While many large US companies in the field tend to have their patent portfolios split roughly 50/50 between US and international patent filings, the top players in China have a combined distribution with well over 75% domestic and only the remainder in international jurisdictions. This means that there is a plethora of technology protected only within the borders of China, and the implications could be significant given the push for AI technology development in China and the wealth of resources available to fuel that development.

So what?

Why does all this matter? When patents are filed in a single jurisdiction only, they are visible to the world and open the door to free use outside the country of filing. In years past, we have seen Chinese companies repurpose Silicon Valley technologies for the China domestic market. With more of a historical patent thicket in the US than in China, this strategy made sense. When development and patent protection have been strong in the US, repurposing that technology in a less protected Chinese market is not only possible, but a viable business model. What we’re seeing now in the emerging field of AI technology, specifically the implementation of such technologies, is the pendulum starting to swing back.

In an interesting reversal of roles, the publication of Chinese patents on technologies not concurrently protected in the US has the potential to drive a copying of Chinese-originated AI tech in the US market. We may see some rapid growth of implementational AI technologies in the US or other western countries, fueled by Chinese development and domestic-focused IP strategy. Of course, there are many other insights to glean out of this wealth of patent data. The use of these patent analytics in the technology space will only increase as the patent offices across the world improve their data reporting and availability. Thanks to advances by some of the major patent offices, visibility into new developments is getting easier and easier. Technology and business intelligence programs stand to gain substantially from the insights hidden in IP data.

Author: Tom Marlow

Source: Oracle

How people from different backgrounds are entering the data science field

Data science careers used to be extremely selective and only those with certain types of traditional credentials were ever considered. While some might suggest that this discouraged those with hands-on experience from ever breaking into the field, it did at least help some companies glean a bit of information about potential hires. Now, however, an increasingly large number of people breaking into the field of data sciences actually aren’t themselves scientists.

Many come from a business or technical background that has very little to do with traditional academic pursuits. What these prospects lack in classroom education they more than make up for with hands-on experience, which has put them in heavy demand when it comes to hire people for firms that need to tackle data analysis tasks on a regular basis. With 89 percent of recruiters saying that they need specialists who also have plenty of soft skills, it’s likely that a greater percentage of outside hires may make it into the data sciences field as a whole.

Moving From One Career to Another

The business and legal fields increasingly require employees to have strong mathematical skills, which has encouraged people to learn various types of skills that they might not otherwise have had. Potential hires who are constantly adding new skills to their personal set and practicing them are among those who are most likely to be able to land a new job in the field of data sciences in spite of the fact that they don’t normally have much in the way of tech industry experience.

This is especially true of anyone who needs to perform analytic work in a very specific field. Law offices who want to apply analytic technology to injury claims would more than likely want to work with someone who has a background in these claims because they would be most capable with the unique challenges posed by accident suits. The same would go for those in healthcare.

Providers have often expressed an interest in finding data analysis specialists who also understand the challenges associated with prescription side-effect reporting systems and patient confidentiality laws. By hiring someone who has worked in a medical office, organizations that are concerned with these rather unique problems posed by these issues. The same is probably true of those who work in precision manufacturing and even food services.

By offering jobs to those who previously handled other unrelated responsibilities in these industries, some firms now say that they’re hiring well-rounded individuals who know about customer interactions as well as how to draw conclusions from visualizations. Perhaps most importantly, though, they’re putting themselves in a better position to survive any labor shortages that the data science field might be experiencing.

Weathering Changes in the Labor Market

While countless individuals naturally always struggle to find their dream job, the market currently seems to be in favor of those who want to transition into a more technically-oriented position. Firms that have to enlarge their IT departments might be feeling the crunch, so creating a resume might be all it takes for someone to land a new job. Since companies and NGOs have to compete for a relatively small number of prospects, it’s making sense for them to hire those who might not have otherwise even thought about working in the tech industry.

Firms that find themselves in this position might not have been able to get anyone to fill these jobs if they didn’t do so. That’s also creating room for something of a cottage industry of data scientists.

The Growth of Non-traditional Data Science Firms

Companies that perform analytics on behalf of someone else are starting to become rather popular. Considering the rise of tracking-related laws, small business owners might look to them as a way to ensure compliance. Anything that they do on behalf of someone else usually has to be compliant with all of these rules per the terms of the agreed upon contract. This takes at least some of the burden off of companies that have little to no experience at all with monetizing their data and avoiding any legal troubles associated with doing so.

While it’s likely that many of these smaller analysis offices will eventually merge together, the fact of the matter remains that they’re growing for the time being. As they do, they’ll probably create any number of additional positions for those looking to break into the data science field regardless of just how far their old careers were from the tech industry.

Author: Philip Piletic

Source: Smart Data Collective