Big Data & Data Science

Watch

• Big Data - Tim Smith ↗TED-Ed
• The most important skills of data scientists | Jose Miguel Cansado | TEDxIEMadrid ↗TEDx Talks
• Beyond the Numbers: A Data Analyst Journey | Anna Leach | TEDxPSU ↗TEDx Talks
• How Data Engineering Works ↗AltexSoft
• The NBA Data Scientist ↗Bloomberg Originals

Read

• Big Data: A Revolution That Will Transform How We Live, Work, and Think ↗This seminal work introduces the foundational concepts of big data, explaining its transformative potential across industries and society.Viktor Mayer-Schönberger and Kenneth Cukier
• Data Science for Business: What You Need to Know about Data Mining and Data-Analytic Thinking ↗An indispensable guide for managers and aspiring practitioners, elucidating the principles of data-analytic thinking crucial for extracting business value.Foster Provost and Tom Fawcett
• Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy ↗A sobering examination of the societal risks inherent in algorithmic decision-making, urging critical scrutiny of data's darker applications.Cathy O'Neil
• The Signal and the Noise: Why So Many Predictions Fail-but Some Don't ↗Explores the art and science of prediction, offering a nuanced perspective on uncertainty and the limits of data-driven foresight.Nate Silver
• Big Data: The Management RevolutionThis influential essay outlines how organisations can harness big data to drive productivity and innovation, reshaping management practices.Andrew McAfee and Erik Brynjolfsson
• The Unreasonable Effectiveness of Data ↗A foundational argument for the power of vast datasets over complex models, challenging traditional approaches to artificial intelligence.Alon Halevy, Peter Norvig, Fernando Pereira

Voices to follow

• Andrew Ng ↗For democratising access to advanced machine learning education and driving practical applications of AI at scale.Co-founder of Coursera, Adjunct Professor at Stanford University, leading figure in AI and machine learning
• Cathy O'Neil ↗Her incisive critiques of algorithmic bias and the societal impact of big data models offer a crucial ethical perspective.Data scientist, author, and activist
• DJ Patil ↗Credited with coining the term 'data scientist' and for his pioneering work in establishing data-driven policy and innovation within government.Former Chief Data Scientist of the United States, venture capitalist

Glossary

• AlgorithmA set of step-by-step instructions or rules that a computer follows to solve a problem or complete a task. Think of it like a recipe for a computer. For example, the steps you follow to search for something on Google are guided by a complex algorithm that decides which results to show you first.
• Artificial Intelligence (AI)The ability of machines to perform tasks that typically require human intelligence, like understanding language, recognizing images, or making decisions. Machine Learning is a big part of how AI works. For example, a smart assistant like Siri or Google Assistant uses AI to understand your voice commands and answer your questions.
• Big DataHuge amounts of information that are so large and complex that regular computer programs can't handle them. It's like trying to drink water from a firehose! For example, all the posts, likes, and shares from every user on Instagram every single day create Big Data.
• Cloud ComputingStoring and accessing data and programs over the internet instead of directly on your computer's hard drive. It's like renting storage and computing power from a big online service. For example, when you save photos to Google Photos or use Google Docs, you're using cloud computing because your files are stored online, not just on your phone or laptop.
• Data AnalystA person who examines data to find useful information, suggest conclusions, and help people make better decisions. They often create reports and visualizations to explain what they found. For example, a data analyst at a sports company might look at sales data to see which types of shoes are most popular in different cities.
• Data ScienceA field that uses scientific methods and computer tools to find patterns and make sense of Big Data. It's about extracting useful knowledge from huge amounts of information. For example, a data scientist might analyze Big Data from a streaming service to figure out which new shows people are most likely to watch next.
• Data SetA collection of related information, like a table or a list, that is organized and stored together. Each row might be one item, and columns are different facts about that item. For example, a list of all students in your school, including their names, grades, and favorite subjects, would be a data set.
• Data VisualizationThe process of presenting data in a graphical or pictorial format, like charts, graphs, or maps, to make it easier to understand and see patterns. For example, a bar graph showing how many students prefer different ice cream flavors is a data visualization.
• Machine LearningA type of Artificial Intelligence (AI) that allows computer systems to learn from data without being explicitly programmed. It means computers can improve their performance on a task over time by looking at more and more examples. For example, when YouTube suggests videos you might like based on what you've watched before, that's machine learning at work.
• Predictive AnalyticsUsing historical data and statistical techniques to forecast future outcomes or trends. It's about trying to guess what will happen next based on what has happened before. For example, a weather app uses predictive analytics to tell you if it's likely to rain tomorrow based on past weather patterns.

Threads 9

Where this connects to other fields, and why it's worth knowing.

• Ritual and Practice Religion

  A scientist once fed pigeons food at totally random times, and they started repeating whatever they happened to be doing when it arrived, like spinning or bowing, as if it caused the food. Computers do the exact same thing: a program can 'learn' a lucky fluke and treat noise as a rule. Both are superstition, a dance built around pure chance.

• Supply, Demand & Price Discovery Economics

  A price tag looks simple, but it's secretly doing a giant math problem. Millions of strangers who never meet, each knowing one tiny thing about supply or demand, all push and pull until the price lands. No single computer or government planner could crunch all that; the market does it without anyone in charge.

• Psychometrics Psychology

  Old personality tests asked you a hundred nosy questions. Now hiring and lending algorithms skip the quiz and just guess your personality from your clicks and likes. Psychology's controversial mind-measuring is back, running silently in the background without your consent or a single form to fill.

• Courts, Judges & Legal Reasoning Law

  When a judge decides a case by looking up the most similar past cases and copying their outcome, they're doing by hand exactly what a simple AI does: 'find the nearest match, predict the same answer.' Which means a legal 'hard case' is just an input that sits far away from anything the judge has seen before, with no close match to copy.

• Genre & Form Literature

  Feed thousands of novels into a computer that tracks whether the mood rises or falls, and something wild pops out: nearly every story fits one of just about six emotional shapes, like 'rags to riches' or 'tragedy.' The endless variety of stories turns out to be as countable as a handful of geometric forms. Data mining found the skeleton hiding inside all our tales.

• Map Scale and Gerrymandering Geography

  Split the same data into different groups and the answer can flip completely: a medicine that looks helpful overall can look harmful once you separate men and women. Mapmakers hit the identical trap when they redraw district lines and change the winner. Same numbers, opposite conclusion, just from how you slice them.

• The Holocaust and Genocide History

  To murder millions, the Nazis first needed to find and sort them, and for that they used early data machines. IBM's punch-card counters let them census people, tag them by category, and track them down. Killing at that scale needed a database before it needed weapons.

• Non-Western Art Arts & Design

  AI image generators learn from huge picture archives that are mostly Western art. So when they create 'art,' they repeat that old bias, quietly baking a Europe-centered idea of what counts as beautiful into the very tools everyone now uses to make new images.

• Economic Systems Economics

  Hayek said a government can never plan an economy well, because prices hold knowledge scattered across millions of people that no planner could ever gather. So here's the modern twist: can big data and AI finally gather it all, or is the problem just too big, even now?