Archive for the ‘Problem Solving’ Category

An interesting story from James Lawther:

…During the second world war over 60 million people were killed.  That was roughly 3% of the world-wide population.  It was a hazardous time.

Among the hardest hit were the bomber crews.  The Eighth Air Force suffered half of all the U.S. Air Force’s casualties.  The British fared as badly.  The chances of surviving the war as a member of the RAF’s bomber command were only marginally better than even.

If flying bombing raids wasn’t dangerous enough, landing when you returned home was also fraught with danger.  Pilots of the Boeing B-17 Flying Fortress had a series of runway crashes, accidentally retracting the landing gear when they touched down.

…Accident investigators blamed these incidents on pilot (or human) error.  There was no obvious mechanical failure.

It wasn’t only Flying Fortresses that had the problem.  There were stories of pilots of P-17 Thunderbolts and B-25 Mitchells making the same mistake.

Nobody would deliberately retract the landing gear when they were still hurtling across the tarmac.  Why the pilots did so was anybody’s guess.  Perhaps the pilot’s attention wandered when they realized they were almost home.

…The authorities asked Alphonse Chapanis,  a military psychologist to explain the behavior.  He noticed that the accidents only happened to certain planes and not others.  There were thousands of C-47 transport planes buzzing about.  Their pilots never suffered from such fatal inattention.

After inspecting the cockpits of the different planes the cause became clear.  On B-17s the controls for the flaps and undercarriage were next to one another.  They also had the same style of handle.   Pilots who retracted the undercarriage when the wheels were on the ground were actually trying to retract the flaps. They just pulled the wrong lever.

In the C-47 the two controls were very different and positioned apart from each other.

The solution: Once he identified the cause Chapanis developed an equally simple solution.  Circular rubber disks were stuck to the levers for the undercarriage and triangles were stuck to the levers for the flaps.  When a pilot touched the rubber, he felt the difference and the crashes stopped…The pilots were well aware of which lever to pull.  It was “human error” that caused the mistake.  But laying the blame on the pilots wasn’t ever going to solve that.


We all make mistakes.  It is in our nature.  Don’t fight it, fix it.




The original post – unfortunately no way to reblog – may be read here.

Read Full Post »

Extracted from an interesting short piece by Christian Madsbjerg is the author of SENSEMAKING: The Power of the Humanities in the Age of the Algorithm:

Silicon Valley needs to get schooled

Silicon Valley is getting antsy. It’s been awhile since we were collectively wowed by the next big thing. The iPhone is ten years old. Uber is eight. The problem isn’t a lack of ideas. As engineers keep breaking new ground, it seems like anything will be possible soon. Why aren’t more of these technologies breaking through to our everyday lives?

What Silicon Valley is missing is an understanding of people—what is meaningful to them, the way they live their day to day lives, what would make a difference for them on an ordinary Tuesday in Phoenix or Shanghai. There is a dearth of deep, nuanced cultural knowledge

From my experience working with major corporations, I would say that technological advancements are only half of the picture. Knowing how to build things is great, but if you have no idea for whom you’re building them—how these inventions will connect with people’s aspirations and challenges—you will fail, no matter how many coding geniuses and data scientists you employ.

If you, like me, are a reader of great novels, you know that almost visceral sensation when you come to understand the world of someone else – the suffering of an Afghan woman, enduring abuse and horrendous conditions to spear her loved ones, or the drab misery of life as an IRS clerk in middle America, someone who had always imagined his life would turn out differently. Literatures—like in-depth journalism, plays, music, art, and even activities like cooking—can put you in the shoes of people unlike you in profound, empathetic way. But the importance of these activities is under attack from the big data-mindset that has invaded both Silicon Valley and many of the world’s biggest corporations.

Spend a few days immersed in a great novel by Tolstoy or with the work of Greek scientist and poet Ptolemy and one is forced to acknowledge that nothing is ever entirely disrupted nor is anything ever completely new. Learning does not function independently of what has come before, but rather in dialogue with it. If executives at Google had taken some time to contemplate this fact, they might have avoided the disastrous rollout to their Google Glass product in 2014. The technology itself functioned just fine. In a narrow Silicon Valley perspective, Google Glass might be considered a successful technology. But when does a piece of technology ever exist independent of a world, a societal structure and culture? Yes, the glasses “worked” but did they belong? Google Glass wearers were dubbed “Glassholes” and people shunned Google Glass wearers at social events. Silicon Valley may have new technology, but in this instance it failed at the much larger challenge of understanding how people relate to one another.

When we use a skill set based in the humanities to understand the world, we gain insight into these deeper issues. And these are the factors that actually drive business forward. Let’s return to China: one by one, the world’s biggest and most cutting edge Silicon Valley companies—Yahoo, eBay, MySpace, Facebook, Twitter, Groupon, and, finally Google—have attempted to develop a meaningful market there. They have come armed with all of the best technical knowledge along with plenty of cash and intellectual property. And yet, today, Internet market leaders in China are still local: Alibaba, Baidu and TenCent.

Technical superiority is a very small part of this story. Limited by their “Silicon Valley” state of mind, American companies simply had no feel for the nuances that made the Chinese marketplace different. With a deeper immersion into the lives of Chinese consumers as well as into their literature, history and religion, technologists might have grasped the more subtle differences between professional and personal network building in Chinese society

When we stop valuing culture, we become blind to the very opportunities that drive “world changing” technology to mass adoption. The greatest challenges and opportunities of the twenty-first century are cultural, not algorithmic. And the greatest tools for the study and understanding of culture exist within the wealth of theories and methodologies that make up the humanities.

To those of you with a liberal arts degree, I say this: your skills are essential in today’s world, and more companies need to recognize that. To those of you with a STEM degree (or who never bothered with college in the first place), I would say: pick up a book or two every month. Go to plays. Travel and immerse yourself in a culture unlike your own.

Without a deep, empathetic understanding of other people, turning that good idea into the next big thing may prove elusive.



The original article may be read here.

Read Full Post »

How a rest stop on the side of the road inspired a web page design

Introducing the YesAQars ago I was driving back to Chicago from Wisconsin. On the Illinois side there are a couple of rest stops over the tollway. It’s a great place to get some gas, grab some caffeine, and stretch your legs a little before the final 50 miles home.

The rest stop usually has a booth where you can buy a iPass so you don’t need to stop and pay tolls all the time. During the day there’s a person in the booth to help answer any questions you have.

It appears that a lot of the same questions are asked over and over. Enough, in fact, that the person who answers them is sick of giving the same answer. That answer is “Yes”.

So they jumped on a computer somewhere and put together what I can only describe as one of the smartest formats for an FAQ I’ve ever seen. A single answer on top, and all the questions below. The answer is always YES!! YES, YES. YES!! Then they taped it to the outside of the booth. You can’t miss it.

Yes Page

I thought this was brilliant. I just love it. Yeah, it’s full of passive aggression and spelling errors and formatting problems, but the idea in itself is so refreshing. It’s folk information art.

Inspired by this, we whipped up our own version of a YES! page for Basecamp 3. It was a fun exercise in messaging and design. We call it the YesAQ.

Yes Page 2

Check it out.


Source: From here.


Read Full Post »

I was camping in a fairly large house, well maintained, surrounded by a number of flowering trees and plants, home to countless birds that treated us to a melodious cacophony announcing their morning foray and home coming in the evening. It was time for the trees to renew themselves – service staff came in the morning and again in the afternoon to sweep off the leaves copiously shed by the tress on the front-yard.  The flowering plants however were still abloom. At times on my touch, a bee would startle me flying out from deep inside the flower.

For one who has lived all his life in Mumbai flats (apartments) where one cannot take ten steps without hitting a wall, one’s auditory nerves constantly assaulted by caw’s of those sullen crows and bark of stray (and house) dogs, this was an overwhelming experience. The spacious front-yard was where I took my mandatory morning and evening walks, my senses enjoying the sights and sounds around.

Get the picture?

The only blot on the scene was the rubble piled up near the neem tree at one corner of the house in the front.  The house owner had not cleared it intending to reuse it in future possibly for patching up parts of the yard.

Yesterday morning, walking near the neem tree I saw a splash of red dried up on the debris. I had not seen it before. Clearly, someone, possibly one of those tradesmen called in for some repair work, had used it as a spittoon after chewing a paan (betel leaf + lime + arca nut shavings + whatever). Unfortunate, but true, in this country one may freely spit in public or even common spaces, but never so within a house. But the perpetrator saw it differently – if the corner was good (?) to pile up the rubble, no one minding, it was ok for him to spit over there.

The ‘Broken Window’ syndrome playing out!


From wiki: ‘Under the broken windows theory, an ordered and clean environment, one that is maintained, sends the signal that the area is monitored and that criminal behavior is not tolerated. Conversely, a disordered environment, one that is not maintained (broken windows, graffiti, excessive litter), sends the signal that the area is not monitored and that criminal behavior has little risk of detection.’

A few broken windows, at times even one, left unfixed for some time is a trigger or invitation for many more, if not all, to be broken.

Much is written on this syndrome as a subject of study under criminology and urban sociology.

Outside of crime, the phenomenon may be observed in many other contexts: projects, product development, organizations, communities and even in personal life.

When a project manager leaves unfixed the first infractions on time deadline, quality issues or team indiscipline…, the first window is broken. His team reads it differently. It’s very likely he would, to his grief, witness many more ‘broken windows’ before long on his way down and out.



Source: wikipedia

Read Full Post »

The context is the recent US elections that belied every prediction made by the media and the ‘elites’. The article by William Davies appearing in recently in The Guardian (here) interestingly attempts an analysis and identifies a social phenomenon that has significance far beyond its immediate context of US elections as you’ll see. As one not knowledgeable or deeply interested to take a stand in US politics, in the extract below I’ve excised author’s political views except where it severely affects the readability to keep the focus on the subject. Also the long linear text is now structured with titles for better understanding of author’s case. Otherwise every attempt is made to preserve the fidelity of author’s thoughts to the extent I understood them. Here we go:

How statistics lost their power…

(the text with in “ ” is edited/recast at places for the same reasons; emphasis made are my doing)

  1. The phenomenon – rejection of Statistics:

In theory, statistics should help settle arguments. They ought to provide stable reference points that everyone – no matter what their politics – can agree on. Yet in recent years, divergent levels of trust in statistics has become one of the key schisms that have opened up in western liberal democracies. Shortly before the November presidential election, a study in the US discovered that 68% of Trump supporters distrusted the economic data published by the federal government. In the UK, a research project by Cambridge University and YouGov looking at conspiracy theories discovered that 55% of the population believes that the government “is hiding the truth about the number of immigrants living here”.

Rather than diffusing controversy and polarisation, it seems as if statistics are actually stoking themNot only are statistics viewed by many as untrustworthy, there appears to be something almost insulting or arrogant about them. Reducing social and economic issues to numerical aggregates and averages blind to local variability seems to violate some people’s sense of political decency.

Nowhere is this more vividly manifest than with immigration. The think-tank British Future has studied how best to win arguments in favour of immigration and multiculturalism. One of its main findings is that people often respond warmly to qualitative evidence, such as the stories of individual migrants and photographs of diverse communities. But statistics – especially regarding alleged benefits of migration to Britain’s economy – elicit quite the opposite reaction. People assume that the numbers are manipulated and dislike the elitism of resorting to quantitative evidence. Presented with official estimates of how many immigrants are in the country illegally, a common response is to scoff. Far from increasing support for immigration, British Future found, pointing to its positive effect on GDP can actually make people more hostile to it. GDP itself has come to seem like a Trojan horse for an elitist liberal agenda. Sensing this, politicians have now largely abandoned discussing immigration in economic terms

The declining authority of statistics – and the experts who analyse them – is at the heart of the crisis that has become known as “post-truth” politics. And in this uncertain new world, attitudes towards quantitative expertise have become increasingly divided. From one perspective, grounding politics in statistics is elitist, undemocratic and oblivious to people’s emotional investments in their community and nation. It is just one more way that privileged people in London, Washington DC or Brussels seek to impose their worldview on everybody else.

  1. Statistics is not elitist:

From the opposite perspective, statistics are quite the opposite of elitist. They enable journalists, citizens and politicians to discuss society as a whole, not on the basis of anecdote, sentiment or prejudice, but in ways that can be validated. We need to try and see them for what they are: neither unquestionable truths nor elite conspiracies, but rather as tools designed to simplify the job of government, for better or worse. The alternative to quantitative expertise is less likely to be democracy than an unleashing of tabloid editors and demagogues to provide their own “truth” of what is going on across society…”

  1. History and motivation for Statistics as a state tool:

“…In the second half of the 17th century, in the aftermath of prolonged and bloody conflicts, European rulers adopted an entirely new perspective on the task of government, focused upon demographic trends – an approach made possible by the birth of modern statistics. Since ancient times, censuses had been used to track population size, but these were costly and laborious to carry out and focused on citizens who were considered politically important (property-owning men), rather than society as a whole. Statistics offered something quite different, transforming the nature of politics in the process

The emergence of government advisers claiming scientific authority, rather than political or military acumen, represents the origins of the expert culture now in disrepute. These path-breaking individuals were neither pure scholars nor government officials, but hovered somewhere between the two. They were enthusiastic amateurs who offered a new way of thinking about populations…Thanks to their mathematical prowess, they believed they could calculate what would otherwise require a vast census to discover.

There was initially only one client for this type of expertise…Only centralised nation states had the capacity to collect data across large populations in a standardised fashion and only states had any need for such data in the first place. Over the second half of the 18th century, European states began to collect more statistics of the sort that would appear familiar to us today. Casting an eye over national populations, states became focused upon a range of quantities: births, deaths, baptisms, marriages, harvests, imports, exports, price fluctuations. Things that would previously have been registered locally and variously at parish level became aggregated at a national level.

New techniques were developed to represent these indicators, which exploited both the vertical and horizontal dimensions of the page, laying out data in matrices and tables, just as merchants had done with the development of standardised book-keeping techniques in the late 15th century. Organising numbers into rows and columns offered a powerful new way of displaying the attributes of a given society. Large, complex issues could now be surveyed simply by scanning the data laid out across a single page.

These innovations carried extraordinary potential for governments. By simplifying diverse populations down to specific indicators, and displaying them in suitable tables, governments could circumvent the need to acquire broader detailed local and historical insight.

  1. Criticisms:

Of course, viewed from a different perspective, this blindness to local cultural variability is precisely what makes statistics vulgar and potentially offensive. Regardless of whether a given nation had any common cultural identity, statisticians would assume some standard uniformity or, some might argue, impose that uniformity upon it.

Not every aspect of a given population can be captured by statistics. There is always an implicit choice in what is included and what is excluded, and this choice can become a political issue in its own right. The fact that GDP only captures the value of paid work, thereby excluding the work traditionally done by women in the domestic sphere, has made it a target of feminist critique since the 1960s. In France, it has been illegal to collect census data on ethnicity since 1978, on the basis that such data could be used for racist political purposes. (This has the side-effect of making systemic racism in the labour market much harder to quantify.)

Despite these criticisms, the aspiration to depict a society in its entirety, and to do so in an objective fashion, has meant that various progressive ideals have been attached to statistics –  ideals of “evidence-based policy”, rationality, progress and nationhood grounded in facts, rather than in romanticised stories…”

  1. Moving from state to political arena and into private hands:

“…The potential of statistics to reveal the state of the nation was seized in post-revolutionary France. The Jacobin state set about imposing a whole new framework of national measurement and national data collection. The world’s first official bureau of statistics was opened in Paris in 1800. Uniformity of data collection, overseen by a centralised cadre of highly educated experts, was an integral part of the ideal of a centrally governed republic, which sought to establish a unified, egalitarian societystatistics played an increasingly important role in the public sphere, informing debate in the media, providing social movements with evidence they could use. Over time, the production and analysis of such data became less dominated by the state. Academic social scientists began to analyse data for their own purposes, often entirely unconnected to government policy goals. By the late 19th century, reformers such as Charles Booth in London and WEB Du Bois in Philadelphia were conducting their own surveys to understand urban poverty.


To recognise how statistics have been entangled in notions of national progress, consider the case of GDP This is fiendishly difficult to get this single number right, and efforts to calculate this figure began, like so many mathematical techniques, as a matter of marginal, somewhat nerdish interest during the 1930s. It was only elevated to a matter of national political urgency by the Second World War, when governments needed to know whether the national population was producing enough to keep up the war effort. In the decades that followed, this single indicator, though never without its critics, took on a hallowed political status, as the ultimate barometer of a government’s competence. Whether GDP is rising or falling is now virtually a proxy for whether society is moving forwards or backwards.

Or take the example of opinion polling, an early instance of statistical innovation occurring in the private sector. During the 1920s, statisticians developed methods for identifying a representative sample of survey respondents, so as to glean the attitudes of the public as a whole. This breakthrough, which was first seized upon by market researchers, soon led to the birth of the opinion polling. This new industry immediately became the object of public and political fascination, as the media reported on what this new science told us about what “women” or “Americans” or “manual labourers” thought about the world

As indicators of health, prosperity, equality, opinion and quality of life have come to tell us who we are collectively and whether things are getting better or worse, politicians have leaned heavily on statistics to buttress their authority. Often, they lean too heavily, stretching evidence too far, interpreting data too loosely, to serve their cause. But that is an inevitable hazard of the prevalence of numbers in public life, and need not necessarily trigger the type of wholehearted rejections of expertise that we have witnessed recently…”

  1. What has changed now to cause resentment?

“… For roughly 450 years, the great achievement of statisticians has been to reduce the complexity and fluidity of national populations into manageable, comprehensible facts and figures. Yet in recent decades, the world has changed dramatically, thanks to the cultural politics that emerged in the 1960s and the reshaping of the global economy that began soon after. It is not clear that the statisticians have always kept pace with these changesEfforts to represent demographic, social and economic changes in terms of simple, well-recognised indicators are losing legitimacy.

Holistic view is no longer adequate:

Consider the changing political and economic geography of nation states over the past 40 years. The statistics that dominate political debate are largely national in character: poverty levels, unemployment, GDP, net migration. But the geography of capitalism has been pulling in somewhat different directions. Plainly globalisation has not rendered geography irrelevant. In many cases it has made the location of economic activity far more important, exacerbating the inequality between successful locations (such as London or San Francisco) and less successful locations (such as north-east England or the US rust belt). The key geographic units involved are no longer nation states. Rather, it is cities, regions or individual urban neighbourhoods that are rising and falling.

The ideal of the nation as a single community, bound together by a common measurement framework, is harder and harder to sustain. If you live in one of the towns in the Welsh valleys that was once dependent on steel manufacturing or mining for jobs, politicians talking of how “the economy” is “doing well” are likely to breed additional resentment. From that standpoint, the term “GDP” fails to capture anything meaningful or credible.

When macroeconomics is used to make a political argument, this implies that the losses in one part of the country are offset by gains somewhere else. Headline-grabbing national indicators, such as GDP and inflation, conceal all sorts of localised gains and losses that are less commonly discussed by national politicians. Immigration may be good for the economy overall, but this does not mean that there are no local costs at all. So when politicians use national indicators to make their case, they implicitly assume some spirit of patriotic mutual sacrifice on the part of voters: you might be the loser on this occasion, but next time you might be the beneficiary. But what if the tables are never turned? What if the same city or region wins over and over again, while others always lose? On what principle of give and take is that justified?

In Europe, the currency union has exacerbated this problem. The indicators that matter to the European Central Bank (ECB), for example, are those representing half a billion people. The ECB is concerned with the inflation or unemployment rate across the eurozone as if it were a single homogeneous territory, at the same time as the economic fate of European citizens is splintering in different directions, depending on which region, city or neighbourhood they happen to live in. Official knowledge becomes ever more abstracted from lived experience, until that knowledge simply ceases to be relevant or credible.

The privileging of the nation as the natural scale of analysis is one of the inbuilt biases of statistics that years of economic change has eaten away.

Classification is not simple: ‘Boxes’ oversimplify

Another inbuilt bias that is coming under increasing strain is classification. Part of the job of statisticians is to classify people by putting them into a range of boxes that the statistician has created (not by respondents): employed or unemployed, married or unmarried, pro-Europe or anti-Europe. So long as people can be placed into categories in this way, it becomes possible to discern how far a given classification extends across the population.

This can involve somewhat reductive choices. To count as unemployed, for example, a person has to report to a survey that they are involuntarily out of work, even if it may be more complicated than that in reality. Many people move in and out of work all the time, for reasons that might have as much to do with health and family needs as labour market conditions. But thanks to this simplification, it becomes possible to identify the rate of unemployment across the population as a whole.

Classification is not simple: Often ‘Boxes’ do not capture intensity

“…Unemployment is one example. The fact that Britain got through the Great Recession of 2008-13 without unemployment rising substantially is generally viewed as a positive achievement. But the focus on “unemployment” masked the rise of underemployment, that is, people not getting a sufficient amount of work or being employed at a level below that which they are qualified for. That is, the intensity of employment is not captured. This currently accounts for around 6% of the “employed” labour force…This is not a criticism of bodies such as the Office for National Statistics (ONS), which does now produce data on underemployment. But so long as politicians continue to deflect criticism by pointing to the unemployment rate, the experiences of those struggling to get enough work or to live on their wages go unrepresented in public debate. It wouldn’t be all that surprising if these same people became suspicious of policy experts and the use of statistics in political debate, given the mismatch between what politicians say about the labour market and the lived realityOpinion polling may be suffering for similar reasons.

Classification is not simple: Respondents create their own ‘boxes’

The rise of identity politics since the 1960s has put additional strain on such systems of classification. Statistical data is only credible if people will accept the limited range of demographic categories that are on offer, which are selected by the expert not the respondent. But where identity becomes a political issue, people demand to define themselves on their own terms, where gender, sexuality, race or class is concerned.

Classification is not simple: Other problems not discussed in the article

Example: ‘Boxes’ may not be not mutually exclusive.


  1. Big Data threatens to damage the ideal of quantitative expertise and its role in political debate

 In recent years, a new way of quantifying and visualising populations has emerged that potentially pushes statistics to the margins, ushering in a different era altogether. Statistics, collected and compiled by technical experts, are giving way to data that accumulates by default, as a consequence of sweeping digitisation. Traditionally, statisticians have known which questions they wanted to ask regarding which population, then set out to answer them. By contrast, data is automatically produced whenever we swipe a loyalty card, comment on Facebook or search for something on Google. As our cities, cars, homes and household objects become digitally connected, the amount of data we leave in our trail will grow even greater. In this new world, data is captured first and research questions come later.

In the long term, the implications of this will probably be as profound as the invention of statistics was in the late 17th century. The rise of “big data” provides far greater opportunities for quantitative analysis than any amount of polling or statistical modelling. But it is not just the quantity of data that is different. It represents an entirely different type of knowledge, accompanied by a new mode of expertise.

Concern on the alignment of the analytics with broader interest of the society:

“… The majority of us are entirely oblivious to what all this data says about us, either individually or collectively. There is no equivalent of an Office for National Statistics for commercially collected big data. We live in an age in which our feelings, identities and affiliations can be tracked and analysed with unprecedented speed and sensitivity – but there is very little to help anchor it in any shared reality in the public interest or public debateIt will fall to the new digital elite to identify the facts, projections and truth amid the rushing stream of data that results less well suited to making the kinds of unambiguous, objective, potentially consensus-forming claims about society that statisticians and economists are paid forWhether indicators such as GDP and unemployment continue to carry political clout remains to be seen

With the authority of statistics waning, and nothing stepping into the public sphere to replace it, people can live in whatever imagined community they feel most aligned to and willing to believe in, bestow up on themselves whatever identity without classification imposed on them; not everything be reliably referred back to some enlightened ideal of the nation state as guardian of the public interest. Where statistics can be used to correct faulty claims about the economy or society or population, in an age of data analytics there are few mechanisms to prevent people from giving way to their instinctive reactions or emotional prejudices…”

Concern about the data and the analytics not being in the public domain:

What is less clear is how the benefits of digital analytics might ever be offered to the public, in the way that many statistical data sets are. Bodies such as the Open Data Institute, co-founded by Tim Berners-Lee, campaign to make data publicly available, but have little leverage over the corporations where so much of our data now accumulates. Statistics began life as a tool through which the state could view society, but gradually developed into something that academics, civic reformers and businesses had a stake in. But for many data analytics firms, secrecy surrounding methods and sources of data is a competitive advantage that they will not give up voluntarily The anonymity and secrecy of the new analytics potentially makes them or whoever has access to them far more politically powerful than any social scientist

A company such as Facebook has the capacity to carry quantitative social science on hundreds of millions of people, at very low cost. But it has very little incentive to reveal the results. In 2014, when Facebook researchers published results of a study of “emotional contagion” that they had carried out on their users – in which they altered news feeds to see how it affected the content that users then shared in response – there was an outcry that people were being unwittingly experimented on. So, from Facebook’s point of view, why go to all the hassle of publishing? Why not just do the study and keep quiet…”

Concern on the potential use of new capabilities to advantageously produce partial truths/untruths for public consumption:

“…What is most politically significant about this shift from a logic of statistics to one of data is how comfortably it sits with the rise of populism. Populist leaders can heap scorn upon traditional experts, such as economists and pollsters, while trusting in a different form of numerical analysis altogether. Such politicians rely on a new, less visible elite, who seek out patterns from vast data banks, but rarely make any public pronouncements, let alone publish any evidence. These data analysts are often physicists or mathematicians, whose skills are not developed for the study of society at all. This, for example, is the worldview propagated by Dominic Cummings, former adviser to Michael Gove and campaign director of Vote Leave. “Physics, mathematics and computer science are domains in which there are real experts, unlike macro-economic forecasting,” Cummings has argued.

Figures close to Donald Trump, such as his chief strategist Steve Bannon and the Silicon Valley billionaire Peter Thiel, are closely acquainted with cutting-edge data analytics techniques, via companies such as Cambridge Analytica, on whose board Bannon sits. During the presidential election campaign, Cambridge Analytica drew on various data sources to develop psychological profiles of millions of Americans, which it then used to help Trump target voters with tailored messaging.

This ability to develop and refine psychological insights across large populations is one of the most innovative and controversial features of the new data analysistechniques of “sentiment analysis”, which detect the mood of large numbers of people by tracking indicators such as word usage on social media, become incorporated into political campaignsIn a world where the political feelings of the general public are becoming this traceable, who needs pollsters?

  1. Conclusion:

“…privacy and human rights law represents a potential obstacle to the extension of data analytics

A post-statistical society is a potentially frightening proposition, not because it would lack any forms of truth or expertise altogether, but because it would drastically privatise them. Statistics are one of many pillars of liberalismThe experts who produce and use them have become painted as arrogant and oblivious to the emotional and local dimensions of politics. No doubt there are ways in which data collection could be adapted to reflect lived experiences better. But the battle that will need to be waged in the long term is not between an elite-led politics of facts versus a populist politics of feeling. It is between those still committed to public knowledge and public argument and those who profit from the ongoing disintegration of those things…”


Read Full Post »

Here’s the latest addition to the lore of what wonders could be wrought through end-point empowerment. Obviously no procedure manual would be able to cover even a small fraction of the large number of field scenarios that occur in real world of customer service. Remains largely unpredictable.

Here we go:

(Lightly edited for readability and conciseness from here – there was no way to reblog the article entirely from its source)

The IndiGo Way of Delighting Customers – A Case Study


“Excuse me, khane mein kya hai?” (“Excuse me, what are the meal options?”), asked the elderly gentleman seated with his wife just one row behind me. The question was directed to an airhostess of an Indigo flight to Pune from Kolkata (India) on a July 2016 evening. All the passengers who had a pre-booked meal, or wanted to purchase on-board, had already been served with their choice of food and beverage, and the cabin crew were busy with cash consolidation and preparing to clean up the deck.

Unlike the passengers around, I was not really taken aback by the loud and out-of-protocol address, as I was already afflicted with the couple’s high pitched conversations in Marathi and Hindi throughout the first hour of the flight. It seemed they were not used to flights. They had even interacted so audibly with their immediate neighbor, an old formally dressed man seating by the aisle seat that I knew they hailed from Satara, returning after spending some time with their newly born grandson at their son’s place at Gangtok (Sikkim). Their son had booked the journey tickets for them, the first leg of which was from Bagdogra to Kolkata, and here they were on their last part of the trip.

“Can I see your boarding pass, Sir?” asked the air hostess politely.

“Here it is”, said the elderly gentleman in a Marathi accented Hindi and extended a card to her.

“Sir, this is the one for the Bagdogra-Kolkata sector, can I please have the pass for this sector?”

To this the man seemed visibly unsettled, searching for the right card with continuous ramblings in Marathi. His wife joined the commotion with “Just see how heartless they are, we haven’t eaten anything since lunch.” The gentleman found the right card and handed it over to the lady in uniform.

As I was finishing my drink over a gripping novel, I paused for a moment to watch the drama happening live beside me.

“Sorry sir, you do not have a meal booked for this sector. You had one in the flight from Bagdogra. However, if you wish, you can now purchase any food or drinks”. The standard pitch.

“Yeh kaise ho sakta hai? Plane mein khana milna hai to? Pehla flight mein bhi diya tha?!” (“How come that’s possible? Planes serve meals, isn’t it? We were served food in the first flight!”), stated the gentleman with a demeanor that said won’t-pay-whatever-hell-comes-up-you-better

The lady excused herself for a quick whisper with her senior, handing over the boarding pass to her.


The lead lady, trained to expect the unexpected, came to the spot in quick time and was straight to the point, “Sir, what would you like to have?”

Seriously, none of the nearby passengers including me was expecting this.

“Dekha? Maine bola tha na?” (“See? I had told you!”), the man said with a smile, oblivious that he was going to receive a free meal. “What do you have in the meals?”

The no-fuss actions that followed next were heart-warming. The lead lady served them 2 sets of sandwiches and mixed fruit beverages with a smile and a wish, “Enjoy your meals, Sir!”.

The couple happily gorged themselves on the food over a high-pitched conversation in Marathi.

I returned to my novel.

Even though the sentences in the book were running in front of my eyes, my mind was absorbed in something else. I was reminded of a talk by Subroto Bagchi, co-founder of Mindtree Ltd…his point was on the right mix of process along with empathy in building and running an organization. All problems of the world can’t be solved by following the right process, unless you have an empathy element to back it up. It becomes particularly important when one deals with the most important aspect of one’s job, people.

If our lead lady had adhered to the laid down process, she would have rightfully refused to oblige the old couple with food packets. That was we had expected out of her. But when she decided to exercise her acumen of empathy, it suddenly made more business sense to all of us…Probably Indigo lost INR 500 (peanuts compared to their daily transactions) as a result, but what they gained was vastly in excess. It satisfied two old people without hassles, averted a possibly ugly scene, created many appreciating passengers, and made me write this blog lauding them.

“Process is not a substitute for building an emotionally rich organization. Process without emotion can quickly bring you down to the lowest common denominator.”

Subroto Bagchi, Co-founder, MindTree Ltd

Let’s not lose sight of the key enabler here: Indigo’s empowerment of its field staff – the end-point delivering the service – that encouraged the lady to make the gesture she did.


Source: Amit Dey, Deputy Manager, Learning & Development | HR at EXL at linkedin.com. And thanks to Anshuman Deshmukh, HR Manager at Genesys International for bringing the article to my notice.

Read Full Post »

And no maths, no equations.

This is from an article that appeared sometime ago edited for readability, deleting (irritating, light-weight and thoroughly avoidable) references to some IT specific code snippets and operations. Here you go:

How the Circle Line rogue train was caught with data

The MRT Circle Line (London underground) was hit by a spate of mysterious disruptions in recent months, causing much confusion and distress to thousands of commuters.

Like most of my colleagues, I take a train on the Circle Line to my office at one-north every morning. So on November 5, when my team was given the chance to investigate the cause, I volunteered without hesitation.

From prior investigations by train operator SMRT and the Land Transport Authority (LTA), we already knew that the incidents were caused by some form of signal interference, which led to loss of signals in some trains. The signal loss would trigger the emergency brake safety feature in those trains and cause them to stop randomly along the tracks.

But the incidents — which first happened in August — seemed to occur at random, making it difficult for the investigation team to pinpoint the exact cause.

We were given a dataset compiled by SMRT that contained the following information:

  • Date and time of each incident
  • Location of incident
  • ID of train involved
  • Direction of train

We started by cleaning the data…

This gave us:

picture-1Screenshot 1: Output from initial processing

No clear answers from initial visualisations

We could not find any obvious answers in our initial exploratory analysis, as seen in the following charts:

  1. The incidents were spread throughout a day, and the number of incidents across the day mirrored peak and off-peak travel times.

picture-2Figure 1: Number of occurrences mirror peak and off-peak travel times.

  1. The incidents happened at various locations on the Circle Line, with slightly more occurrences on the west side.

picture-3Figure 2: The cause of the interference did not seem to be location-based.

  1. The signal interferences did not affect just one or two trains, but many of the trains on the Circle Line. “PV” is short for “Passenger Vehicle”.

picture-4Figure 3: 60 different trains were hit by signal interference.


The Marey Chart: Visualising time, location and direction

Our next step was to incorporate multiple dimensions into the exploratory analysis.

We were inspired by the Marey Chart, which was featured in Edward Tufte’s vaunted 1983 classic The Visual Display of Quantitative Information. More recently, it was used by Mike Barry and Brian Card for their extensive visualisation project on the Boston subway system:

In this chart, the vertical axis represents time — chronologically from top to bottom — while the horizontal axis represents stations along a train line. The diagonal lines represent train movement.

Under normal circumstances, a train that runs between HarbourFront and Dhoby Ghaut would move in a line similar to this, with each one-way trip taking just over an hour:

picture-5Figure 5: Stylised representation of train movement on Circle Line

Our intention was to plot the incidents — which are points instead of lines — on this chart.

Preparing the data for visualisation

With the data processed, we were able to create a scatterplot of all the emergency braking incidents. Each dot here represents an incident. Once again, we were unable to spot any clear pattern of incidents.

picture-6Figure 6: Signal interference incidents represented as a scatterplot

Next, we added train direction to the chart by representing each incident as a triangle pointing to the left or right, instead of dots:

picture-7Figure 7: Direction is represented by arrows and colour.

It looked fairly random, but when we zoomed into the chart, a pattern seemed to surface:

picture-8Figure 8: Incidents between 6am and 10am

If you read the chart carefully, you would notice that the breakdowns seem to happen in sequence. When a train got hit by interference, another train behind moving in the same direction got hit soon after.

What we’d established was that there seemed to be a pattern over time and location: Incidents were happening one after another, in the opposite direction of the previous incident. It seemed almost like there was a “trail of destruction”…

Could the cause of the interference be a train — in the opposite track?

picture-9Figure 9: Could it be a train moving in the opposite direction?

We decided to test this “rogue train” hypothesis.

We knew that the travel time between stations along the Circle Line ranges between two and four minutes. This means we could group all emergency braking incidents together if they occur up to four minutes apart.

We found all incident pairs that satisfied this condition: We then grouped all related pairs of incidents into larger sets…This allowed us to group incidents that could be linked to the same “rogue train”…These were some of the clusters that we identified:

[{0, 1},
{2, 4},
{5, 6, 7},
{8, 9},
{18, 19, 20},
{21, 22, 24, 26, 27},
{28, 29, 30, 31, 32, 33, 34},
{42, 44, 45},
{47, 48},
{51, 52, 53, 56}]

Next, we calculated the percentage of the incidents that could be explained by our clustering algorithm. The result was:

(189, 259, 0.7297297297297297)

What it means: Of the 259 emergency braking incidents in our dataset, 189 cases — or 73% of them — could be explained by the “rogue train” hypothesis. We felt we were on the right track.

We coloured the incident chart based on the clustering results. Triangles with the same colour are in the same cluster.

picture-10Figure 10: Incidents clustered by our algorithm

How many rogue trains are there?

As we showed in Figure 5, each end-to-end trip on the Circle Line takes about 1 hour. We drew best-fit lines through the incidents plots and the lines closely matched that of Figure 5. This strongly implied that there was only one “rogue train”.

picture-12Figure 11: Time of clustered incidents strongly implies that the interference could be linked a single train

We also observed that the unidentified “rogue train” itself did not seem to encounter any signalling issues, as it did not appear on our scatter plots.

Convinced that we had a good case, we decided to investigate further.

Catching the rogue train

After sundown, we went to Kim Chuan Depot to identify the “rogue train”. We could not inspect the detailed train logs that day because SMRT needed more time to extract the data. So we decided to identify the train the old school way — by reviewing video records of trains arriving at and leaving each station at the times of the incidents.

At 3am, the team had found the prime suspect: PV46, a train that has been in service since 2015.

Testing the hypothesis

On November 6 (Sunday), LTA and SMRT tested if PV46 was the source of the problem by running the train during off-peak hours. We were right — PV46 indeed caused a loss of communications between nearby trains and activated the emergency brakes on those trains. No such incident happened before PV46 was put into service on that day.

On November 7 (Monday), my team processed the historical location data of PV46 and concluded that more than 95% of all incidents from August to November could be explained by our hypothesis. The remaining incidents were likely due to signal loss that happen occasionally under normal conditions.

The pattern was especially clear on certain days, like September 1. You can easily see that interference incidents happened during or around the time belts when PV46 was in service.

picture-13LTA and SMRT eventually published a joint press release on November 11 to share the findings with the public.

Final thoughts

When we first started, my colleagues and I were hoping to find patterns that may be of interest to the cross-agency investigation team, which included many officers at LTA, SMRT and DSTA. The tidy incident logs provided by SMRT and LTA were instrumental in getting us off to a good start, as minimal cleaning up was required before we could import and analyse the data. We were also gratified by the effective follow-up investigations by LTA and DSTA that confirmed the hardware problems on PV46.

From the data science perspective, we were lucky that incidents happened so close to one another. That allowed us to identify both the problem and the culprit in such a short time. If the incidents were more isolated, the zigzag pattern would have been less apparent, and it would have taken us more time — and data — to solve the mystery.

Of course, we were most pleased that all of us can now take the Circle Line to work with confidence again.

Daniel Sim, Lee Shangqian and Clarence Ng are data scientists at GovTech’s Data Science Division.



Source: https://blog.data.gov.sg/how-we-caught-the-circle-line-rogue-train-with-data-79405c86ab6a

Read Full Post »

Older Posts »