Coursework
CSci 39542: Introduction to Data Science
Department of Computer Science
Hunter College, City University of New York
Spring 2022

Unless otherwise noted, classwork is submitted via Gradescope. Access information is given during the corresponding lecture.

Due to the internet issues in the lecture hall, for Classwork 2 onward, the classwork will be available until midnight. If you attended class that day, there is an option to earn 0.5 points for attendance and space to include the row and seat number. If you were not able to attend a given lecture, you can still work through the classwork at home and we will replace the fractional point for that classwork with the grade you earned on the final exam. Do not say you were in the room if you did not attend.

• First, the attendance is used in case of contact tracing for covid. The college and health officials need an accurate information for isolation and quarantine planning. You will have to explain your misrepresenting your class presense to the department chair, the dean, and the administrators responsible for student affairs and health, as well as possible isolation for you and your close contacts.
• Second, lying about attendance obtains an unfair advantage and will be submitted to the Office of Student Conduct. It is not worth 0.5 points (that would have been replaced anyway by your final exam score) for a record of academic dishonesty that is kept by both the department and college. The suggested sanction for lying is a 0 on this classwork and the loss of the replacement policy for missed lecture grades. Note: while we suggest a sanction, the final decision about the severity of the sanction is by the Office of Student Conduct.

Classwork 0:   Due midnight, Monday, 31 January.   Available on Gradescope, this classwork focuses on the course syllabus.
If you do have access to the course on Gradescope, write to datasci@hunter.cuny.edu. Include in your email that you not receive a Gradescope invitation, your preferred email, and we will manually generate an invitation.

Classwork 1:   Due 4pm, Monday, 31 January.   Available during Lecture 1 on Gradescope (paper version also available for those without a phone or laptop at lecture), this classwork complements the exploratory data analysis of names and foreshadows the sampling of data in Lecture 2.

Classwork 2:   Due midnight, Thursday, 3 February.   Available during Lecture 2 on Gradescope, this classwork introduces the autograder that is used for the programming assignments. The structure of the sample program mirrors the structure and content of the upcoming Program 1. To get the most out of this exercise, bring a laptop with you to lecture with a development environment (IDE) that has Python 3+ to work through in lecture.
Note: Hunter College is committed to all students having the technology needed for their courses. If you are in need of technology, see Student Life's Support & Resources Page.

Write a function that takes the name of a file and makes a dictionary of the lines of the file.

• make_dict(file_name, sep=': '): Takes a name of a file, file_name and a delimiter sep. The default value is ': '. If a line of the file does not include sep, the line should be ignored. Otherwise, for each line, the string preceding the delimiter sep is the key, and the string after sep is the value. Your function returns the dictionary.

For example, assuming these functions are in a file, cw2.py and run on a file containing names that start with 'A', contacts.txt:

contacts = cw2.make_dict('contacts.txt')
who = 'CS Department'
print(f'Contact info for {who} is {contacts[who]}.')

Contact info for CS Department is 10th Floor HN, x5213.

Another example with nick_names.txt:

nick_names = cw2.make_dict('nick_names.txt', sep = ' ')
names = ['Beth','Lisa','Meg','Greta','Amy','Mia']
for n in names:
    print(f'Full name for {n} is {nick_names[n]}.')

Full name for Beth is Elizabeth.
Full name for Lisa is Elizabeth.
Full name for Meg is Margaret.
Full name for Greta is Margaret.
Full name for Amy is Amelia.
Full name for Mia is Amelia.

If you attended lecture, include the last three lines to the the introductory comment:

"""
Name:  YOUR_NAME
Email: YOUR_EMAIL
Resources:  RESOURCES USED
I attended lecture today.
Row:  YOUR_ROW
Seat:  YOUR_SEAT
"""

Classwork 3:   Due midnight, Monday, 7 February.   Available during Lecture 3 on Gradescope, this classwork asks that you write a program using Pandas and its file I/O. To get the most out of this exercise, bring a laptop with you to lecture with a development environment (IDE) that has Python 3+ to work through in lecture.

Write a program that asks the user for the name of an input CSV file and the name of an output CSV file. The program should open the file name provided by the user. Next, the program should select rows where the field Grade is equal to 3 and the Year is equal to 2019 and write all rows that match that criteria to a new CSV file.

Then a sample run of the program:

Enter input file name: school-ela-results-2013-2019.csv
Enter output file name:  ela2013.csv

School,Name,Grade,Year,Category,Number Tested,Mean Scale Score,# Level 1,% Level 1,# Level 2,% Level 2,# Level 3,% Level 3,# Level 4,% Level 4,# Level 3+4,% Level 3+4
01M015,P.S. 015 ROBERTO CLEMENTE,3,2019,All Students,27,606,1,3.7,7,25.9,18,66.7,1,3.7,19,70.4
01M019, P.S. 019 ASHER LEVY,3,2019,All Students,24,606,0,0.0,8,33.3,15,62.5,1,4.2,16,66.7
01M020,P.S. 020 ANNA SILVER,3,2019,All Students,57,593,13,22.8,24,42.1,18,31.6,2,3.5,20,35.1

Hints:

• Since the Grade column contains a mixtures of numbers (e.g. 3) and strings ("All Grades"), the column is stored as strings.

If you attended lecture, include the last three lines to the the introductory comment:

"""
Name:  YOUR_NAME
Email: YOUR_EMAIL
Resources:  RESOURCES USED
I attended lecture today.
Row:  YOUR_ROW
Seat:  YOUR_SEAT
"""

Classwork 4:   Due midnight, Thursday, 10 February.   Available during Lecture 4 on HackerRank, this classwork introduces the timed coding environment used for quizzes. This classwork mirrors the structure and content of the upcoming Quiz 1. To get the most out of this exercise, bring an electronic device on which you can easily type into a web-based IDE (possible on a phone, but much easier with the bigger screen and keyboards on some tablets and most laptops.
Note: Hunter College is committed to all students having the technology needed for their courses. If you are in need of technology, see Student Life's Support & Resources Page.

Classwork 5:   Due midnight, Monday, 14 February.   Available during Lecture 5 on Gradescope, this classwork focuses on the structure and topics for the optional project, based on the project overview in lecture.

Classwork 6:   Due midnight, Thursday, 17 February.   Available during Lecture 6 on Gradescope, this on-line assignment reviews the different ways to merge DataFrames in Pandas.

Classwork 7:   Due midnight, Thursday, 24 February.   Available during Lecture 7 on Gradescope, this classwork introduces regular expressions for data cleaning. To get the most out of this exercise, bring a laptop with you to lecture with a development environment (IDE) that has Python 3+ to work through in lecture.

Write a program that asks the user for the name of an input HTML file and the name of an output CSV file. Your program should use regular expressions (see Chapter 12.4 for using the re package in Python) to find all links in the input file and store the link text and URL as columns: Title and URL in the CSV file specified by the user. For the URL, strip off the leading https:// or http:// and any trailing slashes (/):

For example, if the input file is:

<html>
<head><title>Simple HTML File</title></head>

<body>
  <p> Here's a link for <a href="http://www.hunter.cuny.edu/csci">Hunter CS Department</a>
  and for <a href="https://stjohn.github.io/teaching/data/fall21/index.html">CSci 39542</a>.  </p>

  <p> And for <a href="https://www.google.com/">google</a>
</body>
</html>
  

Enter input file name: simple.html
Enter output file name:  links.csv

Title,URL
Hunter CS Department,www.hunter.cuny.edu/csci
CSci 39542,stjohn.github.io/teaching/data/fall21/index.html
google,www.google.com

If you attended lecture, include the last three lines to the the introductory comment:

"""
Name:  YOUR_NAME
Email: YOUR_EMAIL
Resources:  RESOURCES USED
I attended lecture today.
Row:  YOUR_ROW
Seat:  YOUR_SEAT
"""

Classwork 8:   Due midnight, Monday, 28 February.   Available during Lecture 8 on Gradescope, this classwork focuses on the GeoJSON format, including hands-on activity with GeoJSON visual editor. To get the most out of this exercise, bring a laptop with you to lecture with a development environment (IDE) that has Python 3+ to work through in lecture.

Classwork 9:   Due midnight, Thursday, 3 March.   Available during Lecture 9 on Gradescope, this classwork is modeled on an analytic reasoning challenge of efficiently computing catchment areas (Voronoi diagrams) for NYC libraries. To get the most out of this exercise, bring a device with you that can access Gradescope's online assignments.

Classwork 10:   Due midnight, Monday, 7 March.   Available during Lecture 10 on Gradescope, this classwork builds intuition about how correlated datasets are. To get the most out of this exercise, bring a device with you that can access Gradescope's online assignments.

Classwork 11:   Due midnight, Thursday, 10 March.   Available during Lecture 11 on Gradescope, this classwork reviews probability distributions and sampling. To get the most out of this exercise, bring a device with you that can access Gradescope's online assignments.

Classwork 12:   Due midnight, Monday, 14 March.   Available during Lecture 12 on Gradescope, this classwork focuses on empirical analysis of random variables. To get the most out of this exercise, bring a laptop with you to lecture with a development environment (IDE) that has Python 3+ to work through in lecture.

Write a function:

• diceSim(D1,D2,trials) that takes as input the number of sides on die 1 (D1) and die2 (D2) and the number of trials. Your function should repeatedly sum pairs of random numbers between 1 and D1 and 1 and D2 and keep track of how many times each sum occurs. The function returns a numpy array with the fraction each sum of rolls occured.

  [0.     0.     0.0259 0.0615 0.0791 0.1086 0.139  0.1633 0.1385 0.114  0.0833 0.0587 0.0281]

If you attended lecture, include the last three lines to the the introductory comment:

"""
Name:  YOUR_NAME
Email: YOUR_EMAIL
Resources:  RESOURCES USED
I attended lecture today.
Row:  YOUR_ROW
Seat:  YOUR_SEAT
"""

Classwork 13:   Due midnight, Thursday, 17 March.   Available during Lecture 13 on Gradescope, this classwork focuses on gradient descent. To get the most out of this exercise, bring a device with you that can access Gradescope's online assignments.

Classwork 14:   Due 4pm, Monday, 21 March.   Available during Lecture 14 on Gradescope, this classwork was a reviewed the first half of the course, following the topics in DS100 Fall 19 Midterm. To get the most out of this exercise, bring a device with you that can access Gradescope's online assignments.

Classwork 15:   Due 4pm, Thursday, 24 March.   Available during Lecture 15 on Gradescope, this classwork focuses on regularization. To get the most out of this exercise, bring a device with you that can access Gradescope's online assignments.

Classwork 16:   Due 4pm, Monday, 28 March.   Available during Lecture 16 on Gradescope, this classwork focuses on computing loss functions. To get the most out of this exercise, bring a device with you that can access Gradescope's online assignments.

Classwork 17:   Due 4pm, Thursday, 31 March.   Available during Lecture 17 on Gradescope, this classwork focuses on nominal data. To get the most out of this exercise, bring a device with you that can access Gradescope's online assignments.

Classwork 18:   Due 4pm, Monday, 4 April.   Available during Lecture 18 on Gradescope, this classwork focuses on linear separability and classification. To get the most out of this exercise, bring a device with you that can access Gradescope's online assignments.

Classwork 19:   Due 4pm, Thursday, 7 April.   Available during Lecture 19 on Gradescope, this classwork focuses on linear algebra. To get the most out of this exercise, bring a device with you that can access Gradescope's online assignments.

Classwork 20:   Due 4pm, Monday, 11 April.   Available during Lecture 20 on Gradescope, this classwork focuses on intrinsic dimensionality. To get the most out of this exercise, bring a device with you that can access Gradescope's online assignments.

Classwork 21:   Due 4pm, Thursday, 14 April.   Available during Lecture 21 on Gradescope, this classwork focuses on distance metrics. To get the most out of this exercise, bring a device with you that can access Gradescope's online assignments.

Using Google Maps API, we generated the amount of time it would take to travel between the following landmarks:

• Bronx Zoo (Bronx),
• Empire State Building (Manhattan),
• National Lighthouse Museum (Staten Island),
• FDR Four Freedoms Park (Roosevelt Island),
• Citi Field (Queens),
• Coney Island (Brooklyn), and
• Hunter College (Manhattan)

Of the three, which best estimates the (aerial) distance?

Classwork 22:   Due 4pm, Monday, 25 April.   Available during Lecture 22 on Gradescope, this classwork focuses on supervised vs. unsupervised learning. To get the most out of this exercise, bring a device with you that can access Gradescope's online assignments.

Classwork 23:   Due 4pm, Monday, 28 April.   Available during Lecture 23 on Gradescope, this classwork focuses on clustering via K-means. To get the most out of this exercise, bring a device with you that can access Gradescope's online assignments.

Classwork 24:   Due midnight, Wednesday, 4 May.   Available during Lecture 24 on Gradescope, this classwork asks your final examination plans (e.g. Do you need a laptop for the coding exam? Would you prefer a left-handed desk for the exam? Do you have accommodations from the Office of Accessability? Do you need to take the exam(s) early?, etc.). If no survey is submitted, the default is that you will take the exams during the regularly assigned times at a right-handed desk and will bring your own laptop to the coding exam.

Classwork 25:   Due 4pm, Thursday, 5 May.   Available during Lecture 25 on Gradescope, this classwork focuses on SQL. To get the most out of this exercise, bring a device with you that can access Gradescope's online assignments.

Classwork 26:   Due 4pm, Monday, 9 May.   Available during Lecture 26 on Gradescope, this classwork is a review for the final examinations. To get the most out of this exercise, bring a device with you that can access Gradescope's online assignments.

Classwork 27:   Due 4pm, Thursday, 12 May.   Available during Lecture 27 on Gradescope, this classwork is on paper at your assigned seat for the exams (or in the front section, if you have requested to take either exam at an alternate time).

Quiz 1: Core Python.   Due 4pm, Friday, 11 February.   Link to access HackerRank available at the end of Lecture 4 (posted on Blackboard).
This first coding challenge focuses on reading and processing data from a file using core Python 3.6+ as in Program 1.

Quiz 2: Pandas Basics.   Due 4pm, Friday, 18 February.   Link to access HackerRank available at the end of Lecture 6 (posted on Blackboard).
This quiz using Pandas and focuses on manipulating and creating new columns in DataFrames as in Program 2.

Quiz 3: Aggregating in Pandas.   Due 4pm, Friday, 25 February.   Link to access HackerRank available at the end of Lecture 7 (posted on Blackboard).
This is the quiz focuses on aggegrating in Pandas as in Program 3.

Quiz 4: Datetime.   Due 4pm, Friday, 4 March.   Link to access HackerRank available at the end of Lecture 9 (posted on Blackboard).
This quiz focuses on aggegrating in Pandas as in Program 4.

Quiz 5: Regular Expressions.   Due 4pm, Friday, 11 March.   Link to access HackerRank available at the end of Lecture 11 (posted on Blackboard).
This quiz focuses on regular expressions in Python as in Program 5.

Quiz 6: Summary Statistics.   Due 4pm, Friday, 18 March.   Link to access HackerRank available at the end of Lecture 13 (posted on Blackboard).
This quiz focuses on computing statistical properties in Python as in Program 6.

Quiz 7: Loss Functions.   Due 4pm, Friday, 25 March.   Link to access HackerRank available at the end of Lecture 15 (posted on Blackboard).
This quiz focuses on computing errors with loss functions in Python as in Program 7.

Quiz 8: Imputing Numerical Values & Fitting Models.   Due 4pm, Friday, 1 April.   Link to access HackerRank available at the end of Lecture 17 (posted on Blackboard).
This quiz focuses on computing errors with loss functions in Python as in Program 8.

Quiz 9: Feature Engineering & Categorical Encoding.   Due 4pm, Friday, 8 April.   Link to access HackerRank available at the end of Lecture 19 (posted on Blackboard).
This quiz focuses on categorical encoding and fitting models in Python as in Program 9.

Quiz 10: Linear Separability & Classifiers.   Due 4pm, Friday, 15 April.   Due to the holiday on April 15, the link to access HackerRank will be available at 8am on Thursday, 14 April to allow students who are observing the holiday to be able to complete the quiz on Thursday.
This quiz focuses on linear separability and classifiers from Classwork 18 and Program 10.

Quiz 11: Dimensionality Reduction.   Due 4pm, Friday, 30 April.   Link to access HackerRank available at the end of Lecture 23 (posted on Blackboard).
This quiz focuses on instrinsic dimensions and dimensionality reduction techniques in Python as in Program 11.

Quiz 12: Clustering.   Due 4pm, Friday, 6 May.   Link to access HackerRank available at the end of Lecture 25 (posted on Blackboard).
This quiz focuses on clustering techniques in Python as in Program 12.

Quiz 13: End-of-Semester Survey.   Due 4pm, Friday, 13 May.   Available on Blackboard at the end of Lecture 27.
This quiz is an end-of semester survey.

All students registered by Monday, 26 January are sent a registration invitation to the email on record on their Blackboard account. If you did not receive the email or would like to use a different account, write to datasci@hunter.cuny.edu. Include in your email that you not receive a Gradescope invitation, your preferred email, and we will manually generate an invitation. As a default, we use your name as it appears in Blackboard/CUNYFirst (to update CUNYFirst, see changing your personal information). If you prefer a different name for Gradescope, include it, and we will update the Gradescope registration.

To encourage starting early on programs, bonus points are given for early submission. A point a day, up to a total of 3 bonus points (10% of the program grade), are possible. The points are prorated by hour. For example, if you turn in the program 36 hours early, then the bonus poins are: (36 hours/3 days)*3 points = (36 hours/72 hours)*3 points = 1.5 points.

To get full credit for a program, the file must include in the opening comment:

• Your name, as it appears in your Gradescope registration.
• The email you are using for Gradescope.
• A list of any resources you used for the program. Include classmates and tutors that you worked with, along with any websites or tutorials that you used. If you used no resources (other than the class notes and textbooks), then you should include the line: "No resources used."

"""
Name:  Thomas Hunter
Email: thomas.hunter1870@hunter.cuny.edu
Resources:  Used python.org as a reminder of Python 3 print statements.
"""

Program 1: Popular Names.   Due noon, Thursday, 10 February.
Learning Objective: to build competency with string and file I/O functionality of core Python.

Program 2: Parking Tickets.   Due noon, Thursday, 17 February.
Learning Objective: to refresh students' knowledge of Pandas' functionality to manipulate and create columns from formatted data.

Program 3: Restaurant Rankings.   Due noon, Thursday, 24 February.
Learning Objective: students can successfully filter formatted data using standard Pandas operations for selecting and joining data.

Program 4: Restaurant Cleaning.   Due noon, Thursday, 3 March.
Learning Objective: to use regular expressions (pattern matching) with simple patterns to filter data from files.

Program 5: Regex Logs.   Due noon, Thursday, 10 March.
Learning Objective: to use regular expressions to parse from log data.

Program 6: Housing Units.   Due noon, Thursday, 17 March.
Learning Objective: to reinforce Pandas skills by aggregating and cleaning to use in map visualiation, and summary statistics methods in Pandas.

Program 7: Housing Model.   Due noon, Thursday, 17 March.
Learning Objective: to enhance on statistical skills and understanding via computation linear regression and loss functions.

Program 8: Yellow Taxi Data.   Due noon, Thursday, 31 March.
Learning Objective: give students practice on implementing model from start to finish and to strengthen understanding of model drift.

Program 9: Logistic Taxi.   Due noon, Thursday, 7 April.
Learning Objective: to train and validate models, given quantitative and qualitative data, as well as assessing model quality.

Program 10: Classifying Digits.   Due noon, Thursday, 14 April.
Learning Objective: to train and validate models, given quantitative and qualitative data, as well as assessing model quality.

Program 11: Digit Dimensions.   Due noon, Thursday, 28 April.
Learning Objective: to increase facility with standard linear algebra approaches and strengthen understanding of intrinistic dimensions of data sets via exploration of the classic digits dataset).

Program 12: EMS Stations.   Due noon, Thursday, 5 May.
Learning Objective: to enhance data cleaning skills and build understanding of clustering algorithms.
Available Libraries: pandas, datetime, numpy, sklearn, and core Python 3.6+.

Program 13: EMS Queries.   Due noon, Thursday, 12 May.
Learning Objective: To reinforce new SQL skills to query and aggregate data.
Available Libraries: pandas, pandasql, and core Python 3.6+.

The grade for the project is a combination of grades earned on the milestones (e.g. deadlines during the semester to keep the projects on track) and the overall submitted program. If you choose not to complete the project, your final exam grade will replace its portion of the overall grade.

Milestones

Deadline:	Deliverables:	Points:	Submission Window Opens:
Monday, 28 February, noon	Opt-In		14 February
Monday, 7 March, noon	Proposal	50	1 March
Monday, 4 April, noon	Interim Check-In	25	14 March
Friday, 29 April, midnight	Complete Project & Website	100	5 April
Monday, 9 May, noon	Presentation Slides	25	14 April
Total Points:		200

Project Opt-In

• Is the final project mandatory?
  No, the final project is optional for this course.
• Will the project be difficult?
  Expect the project to be time consuming because we will hold you to a high standard. However, in turn, we hope that this will produce a high quality project that you could proudly add to your coding portfolio, to showcase when seeking internships and full-time jobs.
• What counts as "opting in" to the project?
  That's easy. Your response to this Gradescope assignment counts as your "opt in". If you respond "No" or do not submit this assignment before the deadline due date, we will count you as having "opted out".
• What happens after I "opt in"?
  If you "opt in", we will continue to send you information on completing the next steps for the final project via Gradescope. If you "opt out", you will no longer receive follow up assignments for the project.
• Does "opting in" place me under obligation to complete the project?
  Yes and no. We would like you to seriously consider your availability before making this commitment. Likewise, we would like to focus our time to help those who are serious about doing this project. That being said, if you decide midway through the process that you no longer have the time nor capacity to complete the project, no harm no foul, your final written exam will once again be weighted 40% of your cumulative grade (see more in the question below).
• How does the final project factor into my final grade?
  If you choose to do the project, your Written Exam will be worth 20% of your overall course grade:
  • Optional Project: 20%
  • Final Exam - Written Exam: 20%
  • Final Exam - Coding Exam: 20%
  If you choose not to do the project, your Written Exam will be worth 40% of your overall course grade:
  • Final Exam - Written Exam: 40%
  • Final Exam - Coding Exam: 20%
  For more details, see the syllabus.

Project Proposal

(50 points)

The window for submitting proposals opens 1 March. If you would like feedback and the opportunity to resubmit for a higher grade, submit early in the window. Feel free to re-submit as many times as you like, up until the assignment deadline. The instructing team will work hard to give feedback on your submission as quickly as possible, and we will grade them in the order they were received.

The proposal is split into the following sections:

• Overview: (10 Points)
  Think of the overview section as the equivalent of an abstract in a research paper or an elevator pitch for the project. The following questions will help you frame your thoughts if you ever have to succinctly describe your project in an interview:
  • Title: should capture the topic/theme of your project.
  • Objective: In 1 to 2 sentences, succinctly describe what you are hoping to accomplish in this project in simple, non technical English.
  • Importance: In 1 to 2 sentences, describe why this project has personal significance to you.
  • Originality: In 1 to 2 sentences, describe why you believe this project idea is unique and original.
  
  
• Background Research: (10 Points)
  In this section, please prove to us that you have already done research in the project you are proposing by answering the questions below.
  • Key Term Definitions: What are some terms specific to your project that someone else might not know? List and define these terms here.
  • Existing Solutions: What are some existing solutions (if any) that are already available for your problem. What are the drawbacks to these solutions?
  
  
• Data: (10 points)
  In order to write a successful proposal, you must already have obtained the data and done basic exploratory analysis on it, enough so that you feel confident you have enough data to answer the questions you wish to explore. We cannot stress this enough: You must use NYC specific data that is publicly available. If your data does not fit this criteria, your proposal will be rejected.

  The following questions will guide you through some criteria you should be using to assess if the data you have is enough for a successful project.

  • Data Source: Include a list of your planned data source(s), complete with URL(s) for downloading. All data must be NYC specific and must be publicly available.
  • Data Volume: How many columns in your dataset? How many rows? If you are joining multiple datasets together, please tell us how many rows and columns remain after the data has been merged into a single dataset.
  • Data Richness: What type of data is in your dataset? You don't need to describe every column. A generalized overview is fine. (e.g. "My data contains 311 complaint types, the date the complaints are created and closed, as well as a description of the complaint"). If you found a data dictionary, feel free to link us to that as well.
  
  
• The Predictive Model: (10 Points)
  A strong data science project should demonstrate your knowledge of predictive modeling. We will be covering models extensively in the latter half of the course. At this stage of the proposal writing, we will not have covered all the modeling techniques yet, so it's okay to be a bit vague here.

  Hint: Look ahead in the textbook at the chapters on "Linear Modeling" and "Multiple Linear Modeling" for the running examples of models.

  • The Predicted (Y): Which column in the dataset are you interested in predicting?
  • The Predictors (X's): Which column(s) in the dataset will be used to predict the column listed above?
  • Python Dependencies: What Python libraries and dependencies will you be using?
  • Security and Privacy Considerations: Will you be working with personal identifiable information (PII)? Can your model be mis-used for evil, not good? If so, how do you plan to mitigate that?
  
  
• The Visualization: (10 Points)
  A key part of making a great data science portfolio are the visualizations. This is a quick and elegant way of showcasing your work during the job hunting process, even to a non-technical audience.

  Thus, a major part of this final project will center around making the following three types of visualizations with the data you choose. If your data cannot support all three types of visualizations, then please, reconsider choosing another dataset.

  • Summary Statistics Plots: Write out in detail at least 3 types of summary statistics graphs you plan to make with your data (e.g. "I plan to make a histogram using the column X").
  • Map Graphs: Write out in detail how you plan to make at least 1 map data visualization using your data (e.g. "I plan to create a choropleth map to visualize the volume of 311 service requests in NYC in 2021").
  • Model Performance Plots: At the time of writing this proposal, we would not have covered how to visualize model accuracy yet. So, no worries if this part is still confusing to you. Give it your best shot on explaining what kind of visualization you think will best showcase that your model is "successful" and "accurate".
  
  

Project Interim Check-In

(25 points)

Instructions can be found on Gradescope, under "2. Project Interim Check-In".

Final Project & Website Submission

(100 points)

Submission Instructions:

• Submit on Gradescope under "3. Final Project & Website Submission".
• Submit your code as one single Python .py file.
• Submit the URL of the website in the introductory comment section of the Python file, preceded by "URL:" so that it can be picked up by the Autograder.
• Submit the code you wrote for your project in the body of the Python file.

For example, for the student, Thomas Hunter, the opening comment of his project might be:

"""
Name:       Thomas Hunter
Email:      thomas.hunter1870@hunter.cuny.edu
Resources:  Used python.org as a reminder of Python 3 print statements.
Title:      My project
URL:        https://www.myproject.com
"""

The Gradescope Autograder will check for the Python file and that includes the title, the resources, and the URL of your website. After the submission deadline, the code and the website will be graded manually for code quality and data science inference.

For manual grading of the project, we are grading for the following:

• Overall Code Quality (code): (25 points)
  We are looking for production level code. No stream of consciousness scripting please. Everything should be in functions, cleanly formatted, and optimized for performance.
  
  
• Good Use of the Data (code & website): (25 points)
  We are looking for evidence that you did analysis on a non-trivial sized dataset and used the data in a meaningful way, drawing from the data cleaning, inference, visualization, and predictive modeling techniques you've learned this semester. Please use the questions in the project proposal and the interim check-in as guidelines for what we are expecting from you.
  
  
• Model Inference (code & website): (25 points)
  We covered a number of predictive modeling techniques in this course, from regularization to linear regression to classification to to SVMs to more. In both the proposal and the interim check in, we have asked you to brainstorm and show us some interim output on the predictive model(s) you have used on your data. Now for the final submission, we ask that you show us 1) the code and output for the final model you chose and 2) the reasoning behind why you chose this model and finally, 3) any inference, interpretation, and insights from the final model results.
  
  
• High Quality Data Visualizations (code & website): (25 points)
  Just like the project interim check-in. We would like to see three types of visualizations: 1) at least one exploratory data analysis graph 2) at least one map related graph and 3) at least one model inference graph. All graphs must be made by you, in Python. Please also make the code available in the Python file submission. If you've already made the visualizations during the interim check-in, we hope that you would take the time to clean them up so that they could be a great show-case of your data visualization skills on your website.
  
  

Presentation Slides

(25 points)

For the last part of the project, include two slides that serve as a graphical overview ("lightning talk" slides) of your project. You should submit to Gradescope, under "4. Project Presentation Slides" a pdf file that contains two slides that summarize your project:

• Slide 1: the front image and title from website, as well as your name
• Slide 2: discoveries & conclusions (with images)

The final exam has two parts:

• Coding Exam: given on the last day of class, Monday, 16 May, 2:45-4pm, in the style of the weekly course quizzes.
• Written Exam: given during finals week on the day and time assigned by the registrar: Monday, 23 May, 1:45-3:45pm.

Logistics:

• The coding exam is administered on HackerRank, using Python 3.6+, libraries used in programming assignments and quizzes, and SQL.
• It is given in two 30-minute parts with a short break in between.
• There are multiple versions of the exam. To access your version, use the passwords on the attendance sheet at your assigned seat.
• To get credit for the exam, you must also turn in the signed attendance sheet before leaving.
• There are assigned seats for the exam that are organized roughly alphabetical by last name.
  • If you submitted a request on Classwork 24 for a borrowed computer, your seat will be up front.
  • If you submitted a request for a left-handed desk, your seat will be on the aisle that leads to the podium ("stage right").
  • For those taking the exams at alternate times (for either exam), you can still participate, and your seat will be up front (so others' locations stay constant for the exams).
  • If you did not submit Classwork 24, we will assume that you are taking the exam at the regular time, using your own laptop, and at a right-handed desk. Your assigned seat is alphabetical by last name.

Exam Rules:

• Leave your Hunter College ID on your desk to be checked during the exam.
• You may have one 8.5"x11" page of notes (content written on front and back).
• The exam must be taken on a laptop or tablet computer (no phones). You may use your own laptop or a department laptop (must be request in advance).
• You may not seek outside help during the exam, including leaving the window once you have started the exam.

Preparing: The exam covers the material covered in lecture and classwork, programming assignments and quizzes, as well as the reading. The coding exam follows the same style as the quizzes. To prepare:

• Bring the laptop (and its charging cables) you plan to use to the review lecture on Thursday, 12 May. Test that the power outlet at your assigned seat works then, so, adjustments can be made before the exam on Monday.
• The past quiz descriptions and solutions are available on Blackboard. Work through all the problems, in quiz-like conditions (limited time, no outside help). Think about variations: what could we ask that's similar? The quiz problems (or parts of them) make excellent exam questions.
• Structure the programming assignments as quizzes: set a 20 minute timer and do each function from the assignments as a quiz. The programming assignments (or parts of them) make excellent exam questions.
• See Lecture 24 for more hints on studying for coding challenges.

Logistics:

• The assigned seating for the written exam will be the same as the coding exam (see above for details).
• To get credit for the exam, you must also turn in the signed attendance sheet before leaving.

Exam Rules:

• Leave your Hunter College ID on your desk to be checked during the exam.
• You may have one 8.5"x11" page of notes (content written on front and back).
• You may not seek outside help during the exam and no electronics are allowed including computers, phones, smart watches, etc.

Format and Preparing:

• The first pages of the exam include the DS 100 Reference Sheet. Read over it as your design your single sheet of notes, since using the reference sheet could open up room for additional information on your notesheet.
• The written exam has 10 multiple-part questions on the material covered in lecture and classwork, programming assignments and quizzes, as well as the reading.
• The questions are similar to those given for past DS 100 exams. Their website contains the exams, as well as solutions. While we used their textbook, our course had slightly different focus (most notably, less statistics), as such not all problems are on their exams fit our course. Here is a list of problems from their exams that are good samples for our exam:
  • Spring 2022 Final: 1a-c,2,4,6a,12
  • Spring 2022 Midterm 2: 1, 6b
  • Spring 2022 Midterm 1: 1a-d, 2-4, 7a, 8
  • Fall 2021 Midterm: 2d, 5-6b, 7
  • Summer 2021 Final: 6a-c, 7, 10a-d, 12a-b, 14c-f, 16e-f
  • Summer 2021 Midterm: 2a-b, 4-9
  • Fall 2020 Final: 2-5a, 9a, 10a,11a
  • Fall 2020 Midterm: 3-5