Faster pages, thumbnail update time, and more!

---

Hello! Here are a few things I've changed/added this week, and a bonus rant at the end!

Improved caching (faster page loading!)

I have improved the service worker's local caching so that it doesn't remove everything whenever I push any change to the website. Now, it should better preserve files that have not changed, leading to fewer network requests and faster page loading.

Some completely static pages (such as the about, extension, and support pages) are now completely cached, which means that no network is required at all to access them! It is not super useful, but I think it's cool :)

Thumbnail update time

I'm sure you've noticed that Whenplane will display when the thumbnail is updated to indicate the show possibly starting soon. It has done this for a while (this feature just turned 1 year old!), but it used to hide the box once the thumbnail was no longer considered 'new'. Whenplane considers thumbnails are <30 minutes old 'new', so that meant that if they took longer than 30 minutes to start the show after changing the thumbnail, it would just disappear off of Whenplane.
This, understandably, caused some confusion.
I have changed the behavior to no longer hide it when it is no longer 'new'; say that it is no longer new. I also added some text to the box to display how long ago the thumbnail was updated.

I actually changed this last week but forgot to announce it. There was also an issue where the time didn't actually display on the first minute, but that should be fixed now.

LTTStore advanced search

Using the new product search system announced last week, I have also added an Advanced Search utilizing some of the features that the new search engine allows. I hope to add more filters and other things over time, but if you have any ideas, please reach out.

Removed products auto-adjust their stock window

Previously, if a product was removed more than 7 days ago, the stock history window would be blank unless you changed the timeframe to something bigger when there was data. Now, Whenplane will automatically adjust this to a time with data. You can still manually change it if you would like.

New Youtube Fetcher

I am also testing a new system for fetching the live stream data from Youtube. It should be much more reliable than any previous system, if it works. I'll keep an eye on it; if it doesn't work, I will roll it back.

If you are curious, I have explained below the issues I have had with the Youtube API in the past, and why those are hopefully over.

The Youtube API - Quotas

The Youtube API has a credit-based quota system. By default, each person gets 10,000 credits per day to make calls to Youtube's API.

That sounds like a lot, right? It should be good for nearly any use case... right?

The caveat is that each request does not cost 1 credit. Some cost hundreds, or even thousands of credits.

The Search endpoint

If you look up how to check if a channel is live using the youtube API, you will find that nearly every result recommends using youtube's Search endpoint. Just give it the channel id you want to check for, filter by livestreams, and you get a list of livestreams.

What is the issue with this? The search endpoint uses 100 credits per request.

That means that if you only want to check if a channel is live, you can only check 100 times per day, which means you can only check once every 15ish minutes.
For whenplane, this 15 minute delay would simply be unacceptable.

In violation

In the past, to get around this issue, I would actually break Youtube API's TOS by using multiple "projects" to effectively get a higher quota allotment. This didn't last forever, and I eventually got an email (actually several, one for each project I used) notifying me that I had to correct my API usage or they would terminate my account.
I corrected this quickly, and found a new solution that could possibly work better.

In grey, kinda

For a new system of having an acceptable polling rate, while trying to stay on Google's good side, after several hours of research, I found a workaround. Instead of using multiple projects to query the search API, I would simply inspect the HTML of the live page. If you didn't know, this page has an interesting quirk. If the channel is not live, then the page will simply show the channel's past livestreams (the "live" tab on the channel). However, if the channel is live, then it shows the live stream instead.
Along with this, the page's canonical would be updated to show if the page was on a live stream, or the past live stream list.

For my new system, I would fetch the HTML of the live page, and check the canonical. If it a link to the channel's stream page, I would know they are not live, and if it was a link to a video page, then I would know that they are live, aswell as the video id of the live stream.

This system worked well for about a week, before Youtube started blocking most of my requests. From then until (hopefully) now, I have had to manually tell the fetcher to use the search API to get the live stream ID.

The best solution (almost) nobody talks about

While looking for a new method for checking live, which I have been doing at least once a week for the past month or so, I came across a Reddit post from someone looking to see the latest video uploaded to a Youtube channel.
They were running into the same issue I was with the Search API, but a helpful commenter had a solution.

Some background details. Every youtube channel has an automatically generated playlist that includes all uploads to that channel. The playlist ID is almost completely identical to the channel's ID, so it is really easy to get if you already have the channel ID (instead of UC at the beginning, its UU).

The comment on this reddit post suggested using the playlist list API, which only costs one credit per request, which is 100x less expensive than the search API.

At this point, an idea formed in my head. What if livestreams are automatically added to this playlist too?

This immediately excited me. Were my pains with checking for livestreams using the Youtube API over??

As fast as possible, I quickly grabbed an API token for testing, threw LTT's uploads playlist id into the playlist list API, and there it was. The WAN shows were included. This was a very good sign.

To further test, I changed the playlist ID to the ID corresponding to Lofi Girl, a channel that has at least one live stream running 24/7. I have regularly used this channel for testing in the past, because I know that they will be live any time I need a live channel for testing.

And sure enough, the current live streams were in this response, too! This is looking more an more promising!

I looked back at my heartrate for this time, and it was actually elevated because of how excited I am. The Youtube API and its quotas has been something I've been struggling with for Whenplane's entire existence (2+ years, and even a little bit before that too) so I was very excited that these struggles might finally be over.

After testing with a random stream I found online, upcoming (scheduled but not yet live) streams showed up there too.

Fingers crossed

Of course, since LTT has not live streamed, I don't know for sure if this works for then, but every indication so far is that it will. I have moved Whenplane to this new method this week. So if it works, you will be able to see it when Whenplane shows that they are live on youtube right away (should be within 10 seconds).

This method of using the playlist API instead of the search API for checking if a Youtube channel is live looks very promising so far. I will update this post if it works today.

Update: It worked perfectly!