Context¶

I used to play a Minecraft competitive community-driven game where matches between teams would be scheduled on Google Sheets.

Why weren't the games scheduled on Google Sheets and not something better like Google Calendar? Since the game was community-driven, power was never centralised at all. The whole community ran on many individuals who would upkeep and maintain stuff for the benefit of the community. Since no one really had any power, if you wanted to do that big of a change you'd need either lots of power from maintaining a shit ton of things (or by being staff and even then, sometimes that wasn't enough), or be really important. I wasn't either lol.

The task here was to make an easy way for the spreadsheet to be read to our Discord server. Of course, people could always check the spreadsheet for themselves but we were developing a bot to make life easier for everyone and maybe to replace an old bot that went offline due to the maintainer stopping updates. Community driven, I told you

In [1]:
from IPython.display import Image
Image('spreadsheet.png')
Out[1]:
No description has been provided for this image

In other words, we had to parse a spreadsheet formatted with times as rows and dates and columns and output correct start times, end times, durations, and dates. I took it upon myself to do this.


Section 0 - How it was done before¶

There was an old bot called wafflebot by redslime (a good friend love you bro) which handled this before. The only issue was that this bot was coded in Java, so we couldn't exactly ctrl c + v everything into our new bot. Along with that, it was a lot more complicated. Wafflebot actually had it's own helpers to get each cell and recreate the spreadsheet as an ArrayList. An example below...

In [ ]:
public static List<Cell> getAllCells() throws IOException {
    List<Cell> cells = new ArrayList<>();
    List<List<Object>> data = getValueRange().getValues();
    for(int rowIndex = 0; rowIndex < MAX_ROW; rowIndex++) {
        for(int columnIndex = 0; columnIndex < MAX_COLUMN; columnIndex++) {
            if(data.size() > rowIndex) {
                if(data.get(rowIndex).size() > columnIndex) {
                    String str = (String) data.get(rowIndex).get(columnIndex);
                    cells.add(new Cell(rowIndex, columnIndex, str));
                } else
                    cells.add(new Cell(rowIndex, columnIndex, ""));
            } else
                cells.add(new Cell(rowIndex, columnIndex, ""));
        }
    }
    return cells;
}

Now, my first language was actually Java. Dunno why I'm telling you this but maybe it gives some insight as to why I didn't want to do a solution like so. I love Java but seeing this monstrosity (sorry lol) made me throw up. I was actually surprised this wasn't done with a third party library and I wanted a nice-looking Pythonic solution. You can read the code yourself if you'd like but I won't go into it too much.


Section 1 - First solution¶

commit 1df9aa1

Now, I kinda knew I wanted to use a 3rd party library to handle all the cell stuff so I could hopefully make the solution look nice. I used the gspread library to import the spreadsheet first of all, and then I stumbled upon the pandas library to actually handle the data.

I knew nothing about pandas, and this was way before AI existed, so I threw myself in the deep end to learn and figure out how to use pandas to make this work.

Now, I attached the spreadsheet above as an image above, but to explain, the first column's first rows were timeblocks of 30 minutes each to represent any time, and the first row's starting columns had dates days to present any date. You could book a time by writing your organised match's name in the spreadsheet, and using caret (this symbol ^) underneath to book any further time needed for the match. New dates would be added by an old player who didn't even play anymore (<3 Hana) as there was only enough space for 22 days in advance. Any matches planned further would be placed in a "TBD" column after the 22nd day.

So I figured the best solution was to isolate each column which represented a day, then add the time column to that column so we would be left with a DataFrame (which is a Pandas object representation of a two-dimensional structure, just like an excel file) with a column of times, then a column of matches with the rows lining up as they were written on the spreadsheet.

I could use this DataFrame to add all matches to a list of my own Matches object which would contain the start time, end time, and name. Afterwards, I'd just print out the list to represent all the matches.

In [ ]:
async def ss(self, ctx, server = "1"):
    gc = gspread.service_account(filename='utils/service_account.json')
    if server == "1":
        values = gc.open_by_key("1CrQOxzaXC6iSjwZwQvu6DNIYsCDg-uQ4x5UiaWLHzxg").worksheet("Upcoming Matches").get("c10:w59")
    else:
        values = gc.open_by_key("1CrQOxzaXC6iSjwZwQvu6DNIYsCDg-uQ4x5UiaWLHzxg").worksheet("Upcoming Matches (Server 2)").get("c10:w59")

A crucial detail I had missed out is that we actually had two servers to play Matches on, so there were two identical spreadsheets for each server. It didn't matter too much, but it just meant that I included an option for the user to enter to view either spreadsheet 1 or spreadsheet 2. I had to do some data cleaning as the spreadsheet contained some hidden rows and columns, and there were rows which were filled with data for people who were to use the spreadsheet. This is self-explanatory, so I won't go too much into detail, but the code above just got the spreadsheet by key.

In [ ]:
df = pd.DataFrame.from_records(values)
row = df.loc[0] #get row with the dates
res = None
if os.name == "nt":
    res = row[row == (date.today().strftime("%#m/%d/%Y"))].index
else:
    res = row[row == (date.today().strftime("%-m/%d/%Y"))].index #find index of todays date, and use that index to start from

I'd first load the spreadsheet into a Pandas DataFrame, and then use today's date to find the column which we must start on.

The first row was retrieved with df.loc[0], and then I had to find the date by comparing strings. First issue

We used American date format (😭😭😭) and I had to remove leading zeros from strftime to match the string to what was on the spreadsheet.

Our bot was hosted on a Linux environment, but a lot of us were on Windows. I'm still not sure why, but the symbol to remove padding is different on each Operating Systems. I used os.name to figure out which OS was used to ensure that the code worked on our machines aswell as on our server.

res was the index of the column where the date was spotted, indicating which column we could start on.

In [ ]:
matches = []
for column in df.iloc[:, res[0] :].columns: #[:, start :] removes the first column. [rows, column]
                                        #remove time column
                                        # if we wanted to make SS past, we would change this to be df.iloc[:, 1:res[0]]
    aDay, aDate = df[column].iloc[1], df[column].iloc[0] #get day and date
    df1 = df.iloc[2:] #remove date/day rows
    day = df1[column] #we iterate through all the days

So we can create our empty matches list, and then finally work in iterating through each day.

Pandas iloc lets us use integer-based indexing. To ignore all previous days, we could simply use this by using our res index to omit all previous days, done like so

df.iloc[:, res[0] :].columns

Then a simple for loop would let us iterate through each column in this new DataFrame.

The spreadsheet contained a row of days (Monday, Tuesday etc), and a further row of Dates. We could get each and save them into a variable, then remove them from the DataFrame using

df1 = df.iloc[2:] #remove date/day rows

This would let us finally iterate through each column full of user submitted data. I named day this column. It would include no time, no date, just a column of user data.

In [2]:
from IPython.display import Image
Image('day_example.png')
Out[2]:
No description has been provided for this image

The blue highlighted section is what our day variable represents. No additional data, just user submitted data.

In [ ]:
day1 = df1[day.astype(bool)].iloc[:, [0, column]] #Remove all cells which dont have a value in them, whilst also adding the time column to it in a new DF
day1 = day1.replace("^", None).ffill() # Then replace all "^" with a cell that doesnt have a value
                                        # So we can use fill forward, which changes 'Dunce ppm, ^ ^' to
                                        # 'Dunce ppm, dunce ppm, dunce ppm'. Ez grouping

I had saved a new variable df1 which now had no date/day rows and only user-submitted data for one column and the column which represented times which was located at positon 0, hence the [0, column]. All the empty were useless, so they would be removed.

Then the first bit of Pandas magic comes in. Since all matches were represented by their name in a cell, then carets below them to represent the durations, I wanted a way to easily figure out the duration. I came across the ffill() function, which propogated entries to the next valid cell. Since I had removed all the empty cells, we could now remove the carets and use ffill() to now replace all the carets with the name of the match they're "attached" to. (INSERT IMAGE)

In [ ]:
first = day1.groupby([column]) # then get a DF from the changes we did ^

This would let us group each cell by identical names, which would make life easier because we can...

In [ ]:
for key, item in first: #Maybe theres a way to do this without iterating at all

Iterate through each group now! This would give us a DataFrame of just those cells. This makes finding the start and end time much easier.

In [ ]:
df2 = item.iloc[[0, -1]] #get the first and last items of the dataframe. This willgive the time it starts/ends
start_time = df2[0].iloc[0].split(" - ")[0] # get the first row (which gives us start time), get the time column and get time
end =df2[0].tail(1).index.item() # same process but for end

item would give us said DataFrame, and then we could use simple iloc indexing for the last and first row.

Since item only has times and the user submitted data, we could get the start time by finding the cell in the first column and first row. The spreadsheet actually included three separate dates - in EST, PST, and GMT. Since there were a lot of Americans (and a lot of them dont understand timezones :p :p), we get the EST time. The end time was achieved by first getting the n number of last rows using tail(n).

In [ ]:
if end == 49: #special case for 11:30pm. we get the next time box which is the alternate method
    end2 = df[0][2] #get the 12am timebox
else:
    end2 = df[0][end+1] #otherwise get the next one along

end2 = end2.split(" - ")[0] #Date

Of course, to get the actual last time, we would need to get the next row. This introduced a weird edge case, where for the last time (11:30pm), we couldn't actually go to the next row to get 12am as it didn't exist. I did a cheap fix by just getting the 12am timebox. I did it this way so we had a robust solution which wouldn't matter if the intervals changed. I didn't want to simply add 30 minutes onto the last timeblock to get the end time incase the spreadsheet ever changed to only have one hour intervals instead. I'll come back to this later

In [ ]:
start = parser.parse(" ".join([aDay, aDate, start_time]), tzinfos={"EST": "UTC-4"}) #add the day, date, and start time to get one datetime
end = parser.parse(" ".join([aDay, aDate, end2]), tzinfos={"EST": "UTC-4"}) #same for the end time

matches.append(Match(key, start, end)) # i made my own class but i dont think its useful, maybe someone else can shorten the code here
matches.sort() #since we made a class of matches, we can now decide how they are compared. Check the Match class, we compare by datetimes

A bunch of parsing later, we finally achieve a DateTime object for the start and end time. We knew the name from the key of which we used groupby with. We could finally add the match into our empty list of Matches.

I actually decided to create a Match object to represent all of the data.

In [ ]:
class Match:
    def __init__(self, name, datetime, end):
        self.name = name
        self.datetime = datetime
        self.end = end

    def date(self):
        return f"{self.datetime.date()}"

    def start_time(self):
        if os.name == "nt":
            return self.datetime.strftime("%#I:%M%p")
        return self.datetime.strftime("%-I:%M%p")


    def end_time(self):
        if os.name == "nt":
            return self.end.strftime("%#I:%M%p")
        return self.end.strftime("%-I:%M%p")

    def __str__(self):
        if os.name == "nt":
            return f"**{self.name}**\n{self.datetime.strftime('%A')}, {self.datetime.strftime('%B')} {self.datetime.strftime('%#d')}\n{self.start_time()} - {self.end_time()} EST\n"
        return f"**{self.name}**\n{self.datetime.strftime('%A')}, {self.datetime.strftime('%B')} {self.datetime.strftime('%-d')}\n{self.start_time()} - {self.end_time()} EST\n"

    def __lt__(self, other):
        return self.datetime < other.datetime

There were a few reasons to create an object to represent the matches.

  • Keep start and end time in DateTime object as long as possible
  • Manage printing the time in a human-readable way with the Windows/Linunx padding differences
  • Possibly expand on the Match object later to save more data and use in other commands.
  • Add a less than method to compare matches for easy use of .sort()

It made the main code pretty and kept all the hidden bs away. I could've gone further and put this in a different class too.

All in all, this made finally printing the matches very simple!

In [ ]:
if matches:
    return await success_embed(ctx, "\n".join(list(map(lambda x: str(x), matches[:7])))) #lambda
await success_embed(ctx, "No upcoming matches")

I could now simply print the matches using the str function on them. All in all, I was very happy with the solution!!!

There were quite a few bugs at first that I let slip, but they were easily fixed and this code has the fixes applied.

This solution wasn't the best though. It made me pretty sad because of a few issues

  • The command only showed spreadsheet 1 or spreadsheet 2, not both. Users who wanted to know the schedule for all matches would have to type the command twice
  • The 11:30pm edge case. It seemed like a lazy fix.
  • Couldn't view the TBD section of the matches. Users deserve to see upcoming matches which didn't have a definitive time/date yet.
  • Completely broke if today's date wasn't found on the spreadsheet - would break if Hana never updated the spreadsheet for more days

I was also unhappy with iterating over the columns. Pandas was powerful, I knew for a fact that there was a way to not iterate.


I was actually deeply unhappy with the solution. I honestly couldn't get it out of my mind: it felt like I had done something so wrong and I needed to improve it.. I didn't have good Pandas knowledge though so whenever I'd have free time I'd constantly google and learn more about Pandas to hopefully come up with a new solution.

Then I found the melt function and everything changed.

Section 2 - The Rewrite¶

commit da4f39d

After being so upset about what I had done I found melt() which could change everything.

Instead of having to iterate through each column, I could melt each column to be one after the other. Hopefully I remembered to attach a diagram of what I'd be doing, but it remove the need for iterating since I could now just group everything.

The only issue was the dates and days. I couldn't melt them aswell. Well, I could, but then I thought why not combine the column representing time, and the rows representing the date? We could have two columns, one with time and date combined, and the other column with user submitted data.

We already had the code for handling the matches in a column. We can use groupby(), we can use ffill(). We could now just apply that to one big column instead, removing the need for iteration.

I've been in contact now with Hana: the maintainer of the spreadsheet. She was a goat and didn't do any unnecessary changes, which made me more comfortable to start using hardcoded integer indexing. We both had the same outlook for the horrendous choice of Google Sheets for a scheduling solution: if it aint broke dont fix it ¯\_(ツ)_/¯. At this point I was FINALLY given edit permissions for the spreadsheet, and a blessing to make the bot handle more.

The code didn't change a lot.

In [ ]:
matches = []
days = df.iloc[0:2, res[0]:22] #if we wanted to make SS past, it would be here
df2 = df.iloc[2:, res[0]:22] #and here

days.iloc[0, :] = days.iloc[0, :] + " " + days.iloc[1, :] #combine days and dates into one row
days = days.iloc[0] #change dataframe into just that one row with days/dates

Days simply represented the first two rows of the sheet. I could combine them into one. It was silly to have two rows representing the same data. Df2 would then all the user-submitted data, excluding the rows of dates and days.

In [ ]:
melted_df = pd.melt(df2) #"melt" all columns into one single column (one after another)
melted_df = melted_df.replace(to_replace=days.index, value=days) #replace numerical index of days to actual date strings

Then the magic of melt came along. We would 'melt' the user-submitted data into just one long column.

In [4]:
from IPython.display import Image
Image('melt.png')
Out[4]:
No description has been provided for this image

A small diagram is shown of what this does. Melt is actually more power than this, but we only needed to do this much. It also gave us an additional index of where each column was, which we replaced with the days.

In [ ]:
time_column = pd.concat([df.iloc[2:, 0]]*len(df2.columns)).reset_index(drop=True) #repeat the times

We could now work on making the first column a representation of both date and time. We had to first start off with repeating the column of times over and over again to match our melted user-submitted data. time_column was now the original time column, just repeated for how ever many days we have which is represented by df2.columns.

In [ ]:
melted_df.iloc[:, 0] = melted_df.iloc[:, 0] + " " + time_column
#then combine days+date with time,
#so our dataframe has columns of [Day, date, time], and [matchname]

Now, it was as easy as adding the times with the days. The comment explains well, we had a DataFrame with the first columns containing [Day, date, time], and another column of [matchname], which I call user-submitted data.

In [ ]:
melted_df = melted_df.replace(["", None], "#~#~#").replace("^", None).ffill()
# 'detect' all events. THIS INCLUDES TIMES WHERE THERES NO MATCHES!

We could then again use ffill(). This time, I actually made it so even empty slots were detected by replacing null-like cells with a special string "#~#~#".

If we ever wanted to let people find available times to book matches, we could simply find the groups which had the special string.

In [ ]:
grouped_df = melted_df.groupby([(melted_df.iloc[:, 1] != melted_df.iloc[:, 1].shift()).cumsum()])
# group by consecutive values

groupby() makes an appearance again in a different way. We'd simply group consecutive values together. I couldn't really use groupby again incase a match with the same name was played at a different datetime. However, I could group by cells below having the same value as the cell above. If the string in the cell ever changed, it'd be in a different group which is desired.

In [ ]:
for group_index, group_df in grouped_df: #got it down to one iteration of just detected events
    #GROUP_INDEX/GROUP_DF REPRESENTS ALL THE GROUPS DETECTED IN THE SS! EVEN EMPTY EVENTS! (where nothing is happening)
    if group_df.iloc[0, 1] == "#~#~#": continue #SO WE REJECT THE EMPTY EVENTS

    match_df = group_df.iloc[[0, -1]]
    start_time = match_df.iloc[0][0].split(" - ")[0]
    index = match_df.index[1]+1
    if index == len(melted_df.index): #case for the last day, last time on SS
        index = match_df.index[1]
    end_time = melted_df.iloc[index][0].split(" - ")[0]
    name = match_df.iloc[1][1]

Now we could simply iterate through each grouped dataframe.

Remember, we included groups of free slots, so we'd need to actually skip by each of them.

However, the code for getting the start and end time was practically the same. We only had one absolute edge case, which was for the match at the last day on the last timeblock, which I was happy with.

Everything was practically the same as the old code before, which is why I'm not explaining it. It works as before!!

And that was done. The rest of the code of adding the matches to a list and printing it out is the same. Nothing actually changed.

This actually fixed the edge case for each day, which was my main concern.

I was happy when I submitted this as it felt like I clearly did something right. I made the way more beautiful, used pandas properly. There was still work to be done, as users would still have to run the command twice to see both spreadsheet 1 and spreadsheet 2... and the spreadsheet being out of date was still going to break everything... but I got through the (what i thought was) the BIGGEST issue.

I actually left out the bugfixes I had to do. If you spotted them, good job. If you didn't... I didn't have to fix a single bug :P

Section 3 - The Small... and the weird Issue?¶

I actually managed to make our bot display both spreadsheets but the change was really simple (commit ce603d5). It just meant opening both spreadsheets instead of one.

We had a further issue. I'm not really sure of the problem anymore because it had been so long ago, but I'm going to explain the issue as best as I could.

Commit 65e0429

Our bot actually specified a timezone in the config file, but we had never used it. I decided it was time to actually use the timezone given in the config file, incase anyone wanted to fork our bot and use it for themselves.

We used the pytz library to get the EST timezone. However, we noticed that the bot was always off by 3 minutes and 58 seconds.

Turns out pytz actually was more powerful that I thought. It contained timezone history, which meant if we got the EST timezone without a date, it'd use the earliest version of the timezone, utc-4:56.02. This was the timezone in use before the 18th of November 1883 btw.

I simply fixed this by using a different library. Could've fixed it by passing in a date so pytz knows I'm trying to use EST as it is now... lol

Section 4 - The Final Touches¶

After this, most of the changes were pretty insignificant. I won't be explaining them too much because I don't feel like there's any importance. The use of Pandas was the main focus on this writeup.

Commit 635cf4c simply added the TBD section by getting the last column of the user-submitted data. The first row in this column had the string 'TBD', so it was very trivial to get this outputted.

Commit 991a760 allowed for more days to be added within the spreadsheet. I believe at this point, someone had finally added more days to the spreadsheet, so it would be updated less frequently.

Commit ec643b6 finally made the bot add the days. Now Hana was freed of her unpaid job of updating the spreadsheet every so often.

Section 5 Lore¶


We made this bot with the idea of people having their own self-hosted copy. Our community was dying as was Minecraft multiplayer. The EULA change Mojang added in 2015 started the death of huge Minecraft servers. No one really disagreed with it, it just made small servers struggle to profit.

Our server was doing fine as besides a few handful of Developers who were employed and salaried by the Owners, everything else was communtiy-ran. We didn't need funding to code these bots as we did it for the love of the community.

We did eventually die too on the 1st of April 2023. I'm not sure how it happened, but Chaploo (our owner (yes THAT fortnite guy)) allowed the code to be distributed within the community, so we now had multiple people running their own versions of the game the community loved. Since our bot was opensourced too, we hoped people would use our code and run the code for their own server.

I've learnt so much from the community so I need to express my gratitude. I love everyone who made the community what it is and I don't think I can ever really repay them all. I learnt Java, we made a whole multi-pov broadcasting channel, and much more. It sucks sometimes that you have to leave things behind, but I have unconditional love for everyone I met there.

I mainly did this write up because I was so invested in solving the one solution of making our bot display the spreadsheet info as neatly as possible. To be honest, it's never left my mind and I honestly feel happy when I think about the days I spent stuck spamming google to learn about Pandas to make a solution I was happy about.

This was my first write up. Never done a write up before. If you know me, you know I really don't leave things half done. I'll improve this writeup eventually and add even more examples, diagrams, and other stuff. Let me think about it lol.