107
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
this post was submitted on 31 Dec 2023
107 points (100.0% liked)
chapotraphouse
13469 readers
782 users here now
Banned? DM Wmill to appeal.
No anti-nautilism posts. See: Eco-fascism Primer
Vaush posts go in the_dunk_tank
Dunk posts in general go in the_dunk_tank, not here
Don't post low-hanging fruit here after it gets removed from the_dunk_tank
founded 3 years ago
MODERATORS
Yeah, my resume right now is Python + SQL (does everything for my job) and a dozen pieces of expensive software that my boss insists we must make use of.
Im starting to use python in everything I can and got to finally get the Python SQL connection work, unfortunately can't get it to work in a jupyter notebook or in an interactive python, so I have to run the scripts, wait 10 minutes and then fix any error or do another step on the process. Funnily enough, is importing the pandas library the bottleneck of the process, the SQL queries are somehow fast in comparison.
I can't help much on the sql connector end (they're black magic and I'm grateful to the sql gods every time they work). But I have a couple of small tips, one if you have CRUD access on the sql server, it's useful to make a small table to use as a way to debug the import script quicker, if no crud access then using top or limit in your sql query would do the trick. Another thing, you may already be doing this, but in your script where you pull data from sql it's generally better to write to a parquet file rather than something like csv. It preserves the pandas dataframe structure which can be slow to cteate.
Also, you can definitely get sql connectors to work in a jupyter notebook, but probably not worth the effort. It's not like on the data pull stage you're doing any exploratory work anyway.
The problem really is that the SQL server doesn't accept connections from my pc, so I had the IT guys install an embedded version of Python in a server that could connect to the SQL server, then from VS code im selecting that Python as the kernel, it's only work as scripting because everytime I run it it use that external Python to run the script in a terminal. On the other hand, the SQL scripts I run them on some Microsoft sql server application, and the Python I just wanted it to do different queries and then process the queries with pandas and export them formatted on excel or wharever.