During the last months of every financial year, there is a tradition, a custom to keep ourselves sane - ranting on tax that we suffered through the year, and any returns we can expect from the taxmen.
In one of the ranting sessions, we had an interesting conversation.
How much tax are we paying?.
What is so interesting?
The tax must be either 25% (new regime) or 30% (old regime).
I use SQL and Python to earn my bread. Being an ETL developer turned Data Engineer, my work is to fetch data from different sources, cleaning it, and making it consumption ready before storing it in some storage space - database, S3 et al.
It’s amusing to know that someone will be consuming this data and pull insights from it like a magician pulling a rabbit off his hat. The reporting tools were the go-to place to churn these processed data. The decision-makers use the reports generated by these tools. The reports are mostly limited to tables, bar, line, or pie charts.
Some reference on how to connect to a Raspberry Pi running in the same LAN network.
I’ve been using Raspberry Pi running in home, for last 10 months. Until a month ago, the sole purpose of the RPi is to filter off the ads using Pi-hole1, then a speed test module was scheduled to run on it. By the setup, the speedtest-cli runs every 10 mins and fetches internet into csv file. the plan is to arrive to some meaningful conclusion of the data.
I’ve been working with Big data since 2017. As I’m from data warehousing background, it was easier for me to understand what’s what, and build an analogy between DWH & big data frameworks. However, the various file formats used in HDFS always caught me off guard.
In DWH, I never considered how the files are stored in DB, it’s managed by the database, maybe DBA might know how its done at the backend, as a DB developer it never bothered me. However, in HDFS we have several types of formats to choose from - Avro, ORC & Parquet. The best way to understand something is to spend time with it. So, decided to see how the different files behave for same data.
I’m working in a corporate firm which employs 100K+ employees. As do with any organization of such a magnitude, there are certain rules & policies which cannot be breached. In most of the occasions, the policies really help us navigate without much of the issue. Yet, there are situations which impedes our progress.
The impedement I felt was merging PDF files. What’s there to worry about, you may wonder. In our org, we have some version of Adobe Reader to view PDFs. Eventhough I’m not a big fan of Adobe, there are no alternatives in my workplace that I know of, at least. The version of Reader I’m allowed to use doesn’t allow me to Merge PDFs. Ironically, it’s necessary to merge the files while submitting the PDFs for various official purpose.