Stats Page - Post your ideas here! - Page 5

Cubert · 05-19-2009, 08:54 PM

just a simple html of your postcount scrolling by flashing in multicolours would beat a 404 page

Tuna Hematoma · 05-30-2010, 09:23 PM

so Goat, how's it hangin?

Space Goat · 05-30-2010, 09:46 PM

It's actually done, I'm just afraid to turn it on, because some of the queries take too long to run, even with all the fancy caching mechanisms I tried to add into the code to help. There have been a few times where I opened certain pages, and the server immediately stopped responding for a few minutes.

Phil Williamson · 05-30-2010, 10:28 PM

is it only certain types of queries that are causing problems? if so, can you just post it with those disabled for the time being? after five years or whatever we just want SOMETHING...

NearHi · 05-31-2010, 07:35 AM

What the hell would a stats page give us!?
"Shaun has posted a million times"
"Tuna has posted 5000 in Bomb this Thread with Posts, another 5000 in the WLA, and 4 times elsewhere"
"AR-Raven has searched Minecraft and Steve Adair 456 times"
???

Kevitus · 05-31-2010, 08:54 AM

Quote:

Originally Posted by Space Goat

It's actually done, I'm just afraid to turn it on, because some of the queries take too long to run, even with all the fancy caching mechanisms I tried to add into the code to help. There have been a few times where I opened certain pages, and the server immediately stopped responding for a few minutes.

Come on you could turn it on for one day at least so we can see if it is really that bandwidth consuming.

Space Goat · 05-31-2010, 01:21 PM

Quote:

Originally Posted by Bryce Farlow

What the hell would a stats page give us!?
"Shaun has posted a million times"
"Tuna has posted 5000 in Bomb this Thread with Posts, another 5000 in the WLA, and 4 times elsewhere"
"AR-Raven has searched Minecraft and Steve Adair 456 times"
???

It's broken up into 12 categories...

Top Movies
Reports the top 10 movies of all time, the past year, month, or week. You can sort by rating, downloads, or views.

Top Reviews
Reports the top 10 reviews of all time, the past year, month, or week. You can sort by replies or views.

It also reports a list of all the people who have rated 10 or more movies, and what their average rating is.

Top Modifications
Reports the top 10 mods of all time, the past year, month, or week. You can sort by rating, downloads, or views.

New Threads
Reports the total number of new threads posted for each month, going all the way back to January 2006.

Most Active Threads
Reports the most active threads of all time, the past year, month, or week. You can sort by replies or views. You can also filter for a specific forum or all forums.

New Posts
Exactly like the "New Threads" report, except it is for posts instead of threads.

Most Active Posters
Reports a list of the most active users (determined by their post count) of all time, the past year, month, or week. You can filter for a specific forum or all forums.

Active Users
Reports a list of all of the users who have logged into their account at least once in the past 60 days. It also reports when they last posted and when they last logged in (unless they set their user CP to make them invisible on the who's online list).

Banned Users
This is just another banned users report like the one we already have.

User Activity
Reports the post count in each forum for individual users. Filter options similar to the reports above can also be applied.

User Ages
Splits the user base into age groups and counts the number of users in each of them. You can also filter for active users, inactive users, or all users.

It also reports the top 10 oldest and youngest users. You have to make your age public in your profile though in order to be included in these top 10s.

User Nationalities
Exactly the same as user ages, except it groups by the nationality flag you have set in your profile. You can also sort by all users, active users, or inactive users.

Quote:

Originally Posted by Kevitus

Come on you could turn it on for one day at least so we can see if it is really that bandwidth consuming.

It's not bandwidth that is the problem. It is CPU and memory usage on the MySQL server that I am concerned about. Some of the reports seem to cause severe spikes, and I've seen the whole site go down temporarily because of it. The "Most Active Posters" report in particular seems to be the worst, because it's basically running a SELECT query on the entire vbulletin_posts table, which has a half a million records in it now. I tried adding a cache mechanism so that after the report is run the first time, it displays a cached copy of the results rather than re-running the query. The cache expires the following day. Even with that, I've still managed to bring the site down by running the report just once during peak hours.

The "security" issues I have always sited are my concerns that a malicious user could come in and just start hammering one of the resource expensive queries until it overwhelms the server and causes a denial of service for other users. The caching mechanism was designed to prevent this, but I haven't got around to adding it to all the reports that need it.

bl0ppin · 05-31-2010, 01:54 PM

Do it then.

Phil Williamson · 05-31-2010, 04:16 PM

yeah please do, that all sounds awesome

Kevitus · 05-31-2010, 08:29 PM

Quote:

Originally Posted by Space Goat

It's not bandwidth that is the problem. It is CPU and memory usage on the MySQL server that I am concerned about. Some of the reports seem to cause severe spikes, and I've seen the whole site go down temporarily because of it. The "Most Active Posters" report in particular seems to be the worst, because it's basically running a SELECT query on the entire vbulletin_posts table, which has a half a million records in it now. I tried adding a cache mechanism so that after the report is run the first time, it displays a cached copy of the results rather than re-running the query. The cache expires the following day. Even with that, I've still managed to bring the site down by running the report just once during peak hours.

The "security" issues I have always sited are my concerns that a malicious user could come in and just start hammering one of the resource expensive queries until it overwhelms the server and causes a denial of service for other users. The caching mechanism was designed to prevent this, but I haven't got around to adding it to all the reports that need it.

You could cronjob the statistics every week and then save the results or something so everything is only executed once a week.

Slime · 06-01-2010, 05:07 PM

I'm confused. I don't know anything about programming, but:

Shouldn't this thing just have stats that are fetched weekly, rather than on the spot? And then you could have it fetch them at a slowed pace rather than how it does it currently, crashing the server because its trying to fetch them quickly enough for the person that entered the search query?
If you just did them weekly, or twice weekly, it wouldn't have to fetch them quickly, AND you could do much more detailed stats because it wouldn't be overloading the server.

I just don't think anybody cares about having stats that pertain to THIS EXACT MINUTE, it would just be cool to have stats updated weekly or so.
I'd personally rather have more detailed stats than live stats.

Phil Williamson · 06-01-2010, 06:43 PM

I'll take anything at this point

Space Goat · 06-01-2010, 10:28 PM

I think you guys are missing the point.

It does not matter when or how often the statistics are fetched, it is where they are fetched from that is the problem. Setting it up to run weekly won't solve anything, because running some of the reports just once is all it takes to bring the server down. Whether you run it on demand or as a scheduled job makes no difference, it still has to run.

The only real way around the problem is to create a statistics table in the database, and populate it as users post and do other activities on the forum that we're interested in deriving metrics from. However, this is a lot more work, and wouldn't even likely be retroactive. I think that would suck and make much of it a lot less meaningful.

The only ones that I'm concerned about is the "Most Active Posters" and "User Activity" reports. I suppose I could just disable those for now, but I personally thought these were two of the most interested reports on the page.

Slime · 06-01-2010, 10:55 PM

And that's why I asked you if its impossible to just have it run slower, doing 1 page at a time since it has an entire week to do so. I figured that would be possible to program since I see PC programs do stuff like this (telling them to index files slower so that you can continue using your PC), but I don't know about web programming.

Space Goat · 06-01-2010, 11:22 PM

No, that's not really an option. The queries in question don't lend themselves to being broken up into pieces. It needs to examine the entire table to get the total post counts.

Funny thing though is that I'm running them right now, and everything seems to be fairly quick. It might have just been the server was unstable last time I was testing this stuff, and just the little bit of extra load was pushing it over the edge.

Space Goat · 06-01-2010, 11:50 PM

OK, the stats page is now live, minus the two reports that were bothering me. Enjoy.

Nixon · 06-01-2010, 11:54 PM

Quote:

Originally Posted by Space Goat

OK, the stats page is now live, minus the two reports that were bothering me. Enjoy.

* pats on the back *

Tuna Hematoma · 06-02-2010, 12:46 AM

I need to rate 4 more movies to get 600!

Slime · 06-02-2010, 12:59 AM

Neat, I like the Ages and Nationalities section.

Phil Williamson · 06-02-2010, 01:03 AM

wheeee how fun thanks! Tuna and I give almost the exact same average rating!!!

Slime · 06-02-2010, 01:22 AM

oh cool, i didn't see that. i give a pretty low rating, wow. im surprised cause ive voted quite a few silly movies 100%

Nixon · 06-02-2010, 01:38 AM

Quote:

Originally Posted by Tuna Hematoma

I need to rate 4 more movies to get 600!

you can go and rate all my movies 100% if you want, i dont mind

James McCloud · 06-02-2010, 01:51 AM

or 0%.

both good choices.

Nixon · 06-02-2010, 02:20 AM

Quote:

Originally Posted by James McCloud

or 0%.

both good choices.

hahaha your one funny guy!.........

05-19-2009, 08:54 PM	#101
Cubert Senior Member Join Date: Jun 2006 Posts: 4,990	just a simple html of your postcount scrolling by flashing in multicolours would beat a 404 page
Cubert Senior Member Join Date: Jun 2006 Posts: 4,990

05-31-2010, 07:35 AM	#105
NearHi Senior Member Join Date: Oct 2001 Posts: 6,483	What the hell would a stats page give us!? "Shaun has posted a million times" "Tuna has posted 5000 in Bomb this Thread with Posts, another 5000 in the WLA, and 4 times elsewhere" "AR-Raven has searched Minecraft and Steve Adair 456 times" ???
NearHi Senior Member Join Date: Oct 2001 Posts: 6,483

05-31-2010, 01:54 PM	#108
bl0ppin Senior Member Join Date: Apr 2005 Posts: 4,644	Do it then.
bl0ppin Senior Member Join Date: Apr 2005 Posts: 4,644

06-01-2010, 05:07 PM	#111
Slime Senior Member Join Date: Oct 2001 Posts: 24,891	I'm confused. I don't know anything about programming, but: Shouldn't this thing just have stats that are fetched weekly, rather than on the spot? And then you could have it fetch them at a slowed pace rather than how it does it currently, crashing the server because its trying to fetch them quickly enough for the person that entered the search query? If you just did them weekly, or twice weekly, it wouldn't have to fetch them quickly, AND you could do much more detailed stats because it wouldn't be overloading the server. I just don't think anybody cares about having stats that pertain to THIS EXACT MINUTE, it would just be cool to have stats updated weekly or so. I'd personally rather have more detailed stats than live stats.
Slime Senior Member Join Date: Oct 2001 Posts: 24,891

06-01-2010, 10:55 PM	#114
Slime Senior Member Join Date: Oct 2001 Posts: 24,891	And that's why I asked you if its impossible to just have it run slower, doing 1 page at a time since it has an entire week to do so. I figured that would be possible to program since I see PC programs do stuff like this (telling them to index files slower so that you can continue using your PC), but I don't know about web programming.
Slime Senior Member Join Date: Oct 2001 Posts: 24,891

05-30-2010, 09:23 PM	#102
Tuna Hematoma Senior Member Join Date: Oct 2001 Posts: 10,055	so Goat, how's it hangin?

05-30-2010, 09:46 PM	#103
Space Goat Administrator Join Date: Sep 2001 Posts: 6,042	It's actually done, I'm just afraid to turn it on, because some of the queries take too long to run, even with all the fancy caching mechanisms I tried to add into the code to help. There have been a few times where I opened certain pages, and the server immediately stopped responding for a few minutes.

05-30-2010, 10:28 PM	#104
Phil Williamson Super Moderator Join Date: Oct 2002 Posts: 16,900	is it only certain types of queries that are causing problems? if so, can you just post it with those disabled for the time being? after five years or whatever we just want SOMETHING...

05-31-2010, 04:16 PM	#109
Phil Williamson Super Moderator Join Date: Oct 2002 Posts: 16,900	yeah please do, that all sounds awesome

06-01-2010, 06:43 PM	#112
Phil Williamson Super Moderator Join Date: Oct 2002 Posts: 16,900	I'll take anything at this point

06-01-2010, 10:28 PM	#113
Space Goat Administrator Join Date: Sep 2001 Posts: 6,042	I think you guys are missing the point. It does not matter when or how often the statistics are fetched, it is where they are fetched from that is the problem. Setting it up to run weekly won't solve anything, because running some of the reports just once is all it takes to bring the server down. Whether you run it on demand or as a scheduled job makes no difference, it still has to run. The only real way around the problem is to create a statistics table in the database, and populate it as users post and do other activities on the forum that we're interested in deriving metrics from. However, this is a lot more work, and wouldn't even likely be retroactive. I think that would suck and make much of it a lot less meaningful. The only ones that I'm concerned about is the "Most Active Posters" and "User Activity" reports. I suppose I could just disable those for now, but I personally thought these were two of the most interested reports on the page.

06-01-2010, 11:22 PM	#115
Space Goat Administrator Join Date: Sep 2001 Posts: 6,042	No, that's not really an option. The queries in question don't lend themselves to being broken up into pieces. It needs to examine the entire table to get the total post counts. Funny thing though is that I'm running them right now, and everything seems to be fairly quick. It might have just been the server was unstable last time I was testing this stuff, and just the little bit of extra load was pushing it over the edge.

06-01-2010, 11:50 PM	#116
Space Goat Administrator Join Date: Sep 2001 Posts: 6,042	OK, the stats page is now live, minus the two reports that were bothering me. Enjoy.

06-02-2010, 12:46 AM	#118
Tuna Hematoma Senior Member Join Date: Oct 2001 Posts: 10,055	I need to rate 4 more movies to get 600!

06-02-2010, 12:59 AM	#119
Slime Senior Member Join Date: Oct 2001 Posts: 24,891	Neat, I like the Ages and Nationalities section.
Slime Senior Member Join Date: Oct 2001 Posts: 24,891

06-02-2010, 01:03 AM	#120
Phil Williamson Super Moderator Join Date: Oct 2002 Posts: 16,900	wheeee how fun thanks! Tuna and I give almost the exact same average rating!!!

06-02-2010, 01:22 AM	#121
Slime Senior Member Join Date: Oct 2001 Posts: 24,891	oh cool, i didn't see that. i give a pretty low rating, wow. im surprised cause ive voted quite a few silly movies 100%
Slime Senior Member Join Date: Oct 2001 Posts: 24,891

06-02-2010, 01:51 AM	#123
James McCloud Senior Member Join Date: May 2005 Posts: 4,214	or 0%. both good choices.