After my great internship experience, I decided to join back full-time with Two Sigma!
Unfortunately, I cannot share much about the work that I did at Two Sigma as we dealt with a lot of secure IP, but here’s the gist. I worked as a data engineer at the company on a team called Foundational Data Engineering (FDE). On FDE, we processed various types of fundamental datasets including different asset classes like stocks, bonds, options, futures, and other derivatives. Reference data and pricing data for these assets are equally important for the company to train, simulate, and eventually deploy models in production. These datasets need to be appropriately parsed, arbitrated, and cleaned into a company-wide agreed format since the data comes from lots of different vendors and the data can be used by the multiple consumers within the company.
I helped maintain and migrate older data pipelines to newer architectures by using tools such as Ibis and Pandas to manipulate and serve large datasets in BigQuery. This architecture became one of the newer standards in the team. As a result, I became quite comfortable with SQL in general while working with both SQL Server and BigQuery during my stint at Two Sigma.
