In this assignment, students will work on a dataset to answer specific exploratory questions by applying one or more techniques seen in class: supervised learning, recommender systems, unsupervised clustering, frequent itemset mining, data stream analytics, graph analysis, and similarity search. Students will implement the project in Python, using Jupyter notebooks and a data analytics library among Apache Spark, Dask or scikit-learn.

As a participating organization, you’ll be asked to provide a particular dataset and a first set of related questions to be answered by the team using the dataset.

The expected project milestones are as follows:

1. Project definition: students will summarize the project, including: (1) the dataset of interest, (2) the set of exploratory questions to be answered with the dataset, using techniques studied in class.
2. Model design: students will choose a class of models in {supervised learning, recommender systems, unsupervised clustering, frequent itemset mining, data stream analytics, graph analysis, similarity search}. They will outline how the data model could be applied to the dataset to answer the exploratory question(s). They will research algorithms and techniques to implement this class of model.
3. Data preparation: students will inspect the dataset, identify missing data, outliers, data types (categorical data in particular), and write Apache Spark or Dask programs to correct for potential issues.
4. Model implementation: students will implement the model with Apache Spark, Dask or scikit-learn.
5. Model evaluation: students will identify evaluation metrics for the model, implement, and discuss them.