`{col_rename_map.get(old_key): val for key, val in config.values()}` where `col_rename_map` is the dict mapping the two as you suggested…
Assuming I’ve understood what you’re asking, which I’m not totally sure.
Had this discussion the other day and figured I’d throw it to the masses to get thoughts on the best/most pythonic way of approach.
Need to map old column names to new column names from a json config.
My thoughts are iterate over a dict with ‘’’ {“old_col_name”:”new_col_name”}’’’ And access as ‘’’for k,v in dict.items() Df.with_columns(k).alias(v)’’’
Colleague things this isn’t clear enough and should be a list of dicts with explicit keys
‘’’ [{“old_col_name”:”old_col_value” “New_col_name”:”new_col_value”}]’’’
And the access as
‘’’for dict in list_of_dicts: Old_col = dict[“old_col_name”] New_col = dict[“new_col_name”]’’’
I’ve got a good few reasons why I think mine is the better option but thought I’d get some other opinions to see if I’m missing anything obvious? Which would you choose and why?
`{col_rename_map.get(old_key): val for key, val in config.values()}` where `col_rename_map` is the dict mapping the two as you suggested…
Assuming I’ve understood what you’re asking, which I’m not totally sure.
renames = [("old1", "new1), ("old2", "new2")] for old_name, new_name in renames: my_dict[new_name] = my_dict.pop (old_name)
Yours is obviously better and perfectly appropriate. In a *mapping* of old to new column names, you can directly access the new column name of any old column name. In a list of dicts, you cannot do that, and must iterate the list before you can do anything. For no apparent benefit. You use a list of dicts like that if keys may be duplicated. Which is apparently not the case here. So there’s zero benefit to that extra wrapper, and in fact only drawbacks.
It really depends, if you want to have an UI to do the mapping the list of dicts can be a good choice
Yeah I think there’s a solid argument for renaming the variable to a clearer intent which would solve most of the issue.
You should obviously create a RenameColumnsBase class as an abstraction for the concept of renaming columns, with multiple construction helpers from_dict, from_list_of_dicts, etc and then implement the renaming in a RenamePolarsColumns subclass. /s It looks like you're using polars. Why not just match the df.rename() function's signature?
Yeah, in Pandas I would just go df.rename(columns={“old\_col\_name1”:”new\_col\_name1”,“old\_col\_name1”:”new\_col\_name2”})
You forgot to work a microservice into that mess!
We aren’t using rename as it’s to copy the column while leaving the original intact as a reference of the unmodified data. You’re actually not far off with the structure actually. That logics wrapped in a rule class that’s build using a strategy pattern as a custom data cleaning recipe for each of our product suppliers.
The alternative doesn't make sense at all. What is old\_col\_value (and the new one) supposed to be? The only sensible way of thinking it to map from an old name to a new name. But you don't need to iterate through the dict explicitly. Both pandas and polars have a df.rename method. Just pass the dictionary to it.
Part of that’s a my bad in the explanation. Typing it in a rush this morning. Mea culpa!
Wow. Just wow...
When you have a list of dicts that all have the same keys, that's the same damn thing as a dataclass. And this problem doesn't need dataclasses.
Yeah that was kinda my thought, seems to be over engineered for something that can be explicitly understood from the code? You’d could use data classes etc but seems like it’s not complicated enough to justify adding another layer to?
I'd pick typed for the values as well, and define two TypedDicts, with a function (ideally a classmethod on the second one) that maps from one to the other.
I’ve got a good few reasons why I think mine is the better option
Yet you didn't list a single one?
Your two pieces of code don't do the same thing, so I'm not sure how you are comparing them. I would probably also do something closer to the first piece in my own code.
However, the most important point is this:
Colleague things this isn’t clear enough
You don't work with Reddit users. You work with your colleague. It doesn't matter what some random guy on the internet tells you they prefer because you have no interaction with them outside of this thread. It does matter a ton what the people you directly work with think.
Do you need me to list them for you to make your own decisions? The two pieces of code are basically the same, both access two column names. I’m not sure how they are really different unless you need me to explicitly state that list of dicts also uses a polars method after? Mate, I’m the senior engineer on this. He’s a junior, I’ve already made my position clear on this to him. I put this out there purely out of interest and to sense check myself. But thanks for the unhelpful comment 🙄
Sounds like youre using spark? If so, use withColumnsRenamed, which renames multiple columns in a single operation instead of looping multiple operations for the same output
Ah polars, not actually something I use a lot but the performance improvements justified the swap from pandas for this service. Also spark background so the syntax is a bit more comfortable
[removed]
Even if your colleague POV made some sense (it doesn't), it's SO innefficient. Why create a list of HASHED map for each elements instead of a tuple????
Sounds like adding a comment for your colleague should sort this “mess”.
Just go reach for pandas and have fun...
small wrinkle on the dict version that hasn't come up: if that json ever ends up with the same old column listed twice, python's json module just takes the last one and says nothing. json.loads('{"a": 1, "a": 2}') gives you {'a': 2}, no error. wouldn't change which shape i'd pick, but it's worth asserting the mapping length against something when you load the config, because a fat-fingered duplicate in there is completely silent.
No matter the comments here the argument will not be settled. Your goal with 5he post is to be told “you’re right”. You coworker is likely to giveThis thread less weight than they gave their senior engineer. If you do take this to them saying “see…I’m right” it will do nothing more than drive a wedge between you and create conditions where this is likely to repeat. As the senior you should break the stalemate by recognizing there are many valid ways to solve the problem and the priority should be on moving forward rather than winning a pissing match that is un winnable.