Solving Problems Using Code
Posted by Aly Sivji in Quick Hits
This canonical URL for this blog used to be alysivji.github.io. I have also set up nginx to forward requests to alysivji.com and sivji.com to this blog.
This is all well and good when are you running nginx out of a VM, but what if you are …
Posted by Aly Sivji in Quick Hits
Need to access a private Git repo inside of a Dockerfile during the docker build process.
The main idea is to use HTTP Basic Authentication while downloading files using the git command line client.
Create a Personal Access Token with repo scope using your Git provider's interface …
Posted by Aly Sivji in Quick Hits
Yesterday I gave an abridged version of my Data Science Workflows using Docker Containers talk at my employer's Biweekly Engineering Lunch session.
It had been a minute since I last gave this talk so I ran through the examples to make sure everything worked. Having all my examples containerized is …
As a learning exercise, I migrated all my side projects to Kubernetes. I was using Fluent Bit to forward application container logs to my ElasticSearch instance.
Everything was working as expected, but the cluster was starting to get expensive. DigitalOcean Kubernetes is competitively priced, but paying $80/month to run …
Posted by Aly Sivji in Quick Hits
I recently purchased a new computer and needed to transfer files from the old computer to the new one. Tried a USB drive, no luck. Tried copying files over the network via the macOS Finder, but the sheer number of files (~500,000) made this an exercise in futility.
Googling …
I started a new job in October and bought a 2018 MacBook Pro to celebrate. Purchased the 13" Touchbar with (mostly) maxed out components: 2.7GHz i7, 16GB RAM, and 1TB SSD. It's a huge step up from my late-2013 model; ~6x according to Moore's Law.
As soon as I …
A Python 3.6 feature that has flown under the radar is the revamped email package.
In this post we will explore email and walk through an example of creating and sending an email using the Python Standard Library.
emailThe email package is comprised of three major components …
Posted by Aly Sivji in Quick Hits
In Python, functions are objects. This means we can return them from other functions.
In this Quick Hit, we will use this property of functions to mock out an external API with fake data that can be used to test our internal application logic.
Note: I previously used Python functions to simulate the behavior of a case statement
Posted by Aly Sivji in Data Analysis
In this post, I will describe how to use BSON-NumPy to pull data out of Mongo and into pandas. While this library is still in the prototype stage, it's hard to to ignore the 10x speed improvement that comes from reading BSON documents directly into NumPy.
For this example, I will be analyzing a collection of my tweets
Posted by Aly Sivji in Quick Hits
I was floored when I discovered I could use tuples (and namedtuples) as Python dictionary keys.
Turns out this is not totally correct.
Started reading Fluent Python and learned that tuples are not always immutable. For instance, they can contain list elements...
Posted by Aly Sivji in Quick Hits
ChiPy hosts Python Project Night on the 3rd Thursday of every month. Groups of four are given an hour to work through a structured exercise with each person spending 15 minutes at the keyboard writing code. At the end of the hour, one group is selected to present their solution.
I love using the Command Line; it's my favourite part of programming. It wasn't always this way. I used to worry that the commands I entered would destory the delicate equilibrium where everything just worked. Would this be the pip install that ended it all?
Enter virtual environments.
A virtual …
Summary
- Explore Plotly's new Dash library
- Discuss how to structure Dash apps using MVC
- Build interactive dashboard to display historical soccer results
I spent a good portion of 2014-15 learning JavaScript to create interactive, web-based dashboards for a work project. I wrapped D3.js with Angular directives to create modular …
Summary¶
- Explore
withstatements and the context manager protocol- Implement context manager class to query MongoDB
- Convert
try...finallyblock towithblock and increase code readability
I recently read Steve McConnell's Code Complete to level up my software development skill-set. The book has helped me become more deliberate about programming and problem solving in general. Before I sit down to write a single line of code, I take some time to plan out the work I am going to do versus code by the seat of my pants
Posted by Aly Sivji in Quick Hits
Was reading the Python design FAQs and saw that immutable elements can be used as dictionary keys.
Wait a minute.... tuples are immutable, can they be used as dictionary keys? YES!
(Note: This post is part of my reddit-scraper series)
Summary
- Running Scrapy spider as a script
- Scheduling script to run on PythonAnywhere cloud
Previously on Siv Scripts, we created a web scraping pipeline to pull Top Posts from Reddit and store them in a MongoDB collection. At this stage, we …
(Note: This post is part of my reddit-scraper series)
Summary
- Web frameworks intro
- Explore the Flask microframework
- Understand the Model-View-Controller (MVC) pattern
- Build Flask web app
Last time we started our web application adventure by learning how to generate dynamic HTML webpages from data stored in MongoDB using MongoEngine and …
(Note: This post is part of my reddit-scraper series)
Summary
- Overview of MongoDB
- Discussion of Object-Relational Mapping (ORM)
- Use MongoEngine to get items out of MongoDB
- Render HTML pages using Jinja2
- Interact with REST API to send emails with Requests
Previously on Siv Scripts, we implemented a web scraping pipeline …
Posted by Aly Sivji in Data Analysis
Summary
- Use Python to display location data on interactive map
- Cluster points in a heatmap to visualize usage
One of my favorite things about the Python programming language is that I can always import a library to do the heavy lifting and focus my attention on application logic. It's a …
Summary
- Discuss advantages of using Scrapy framework
- Create Reddit spider and scrape top posts from list of subreddits
- Implement Scrapy pipeline to send scraped data into MongoDB
Wouldn't it be great if every website had a free API we could poll to get the data we wanted?
Sure, we could …