3dmm.com

3dmm Chatroom: Daily meetings at 11pm GMT (6pm EST)
Go Back   3dmm.com > 3dmm.com > Administrative Contact
User Name
Password
Register Site Rules FAQ Members List

Reply
 
Thread Tools Display Modes
Old 05-19-2009, 08:54 PM   #101
Cubert
Senior Member
Cubert's Avatar
Join Date: Jun 2006
Posts: 4,990
just a simple html of your postcount scrolling by flashing in multicolours would beat a 404 page
Cubert is offline   Reply With Quote
Old 05-30-2010, 09:23 PM   #102
Tuna Hematoma
Senior Member
Tuna Hematoma's Avatar
Join Date: Oct 2001
Posts: 10,055
so Goat, how's it hangin?


Tuna Hematoma is offline   Reply With Quote
Old 05-30-2010, 09:46 PM   #103
Space Goat
Administrator
Space Goat's Avatar
Join Date: Sep 2001
Posts: 6,042
It's actually done, I'm just afraid to turn it on, because some of the queries take too long to run, even with all the fancy caching mechanisms I tried to add into the code to help. There have been a few times where I opened certain pages, and the server immediately stopped responding for a few minutes.
Space Goat is offline   Reply With Quote
Old 05-30-2010, 10:28 PM   #104
Phil Williamson
Super Moderator
Phil Williamson's Avatar
Join Date: Oct 2002
Posts: 16,900
is it only certain types of queries that are causing problems? if so, can you just post it with those disabled for the time being? after five years or whatever we just want SOMETHING...


Phil Williamson is offline   Reply With Quote
Old 05-31-2010, 07:35 AM   #105
NearHi
Senior Member
NearHi's Avatar
Join Date: Oct 2001
Posts: 6,483
What the hell would a stats page give us!?
"Shaun has posted a million times"
"Tuna has posted 5000 in Bomb this Thread with Posts, another 5000 in the WLA, and 4 times elsewhere"
"AR-Raven has searched Minecraft and Steve Adair 456 times"
???


NearHi is offline   Reply With Quote
Old 05-31-2010, 08:54 AM   #106
Kevitus
Senior Member
Kevitus's Avatar
Join Date: Jan 2003
Posts: 4,542
Quote:
Originally Posted by Space Goat
It's actually done, I'm just afraid to turn it on, because some of the queries take too long to run, even with all the fancy caching mechanisms I tried to add into the code to help. There have been a few times where I opened certain pages, and the server immediately stopped responding for a few minutes.

Come on you could turn it on for one day at least so we can see if it is really that bandwidth consuming.


Kevitus is offline   Reply With Quote
Old 05-31-2010, 01:21 PM   #107
Space Goat
Administrator
Space Goat's Avatar
Join Date: Sep 2001
Posts: 6,042
Quote:
Originally Posted by Bryce Farlow
What the hell would a stats page give us!?
"Shaun has posted a million times"
"Tuna has posted 5000 in Bomb this Thread with Posts, another 5000 in the WLA, and 4 times elsewhere"
"AR-Raven has searched Minecraft and Steve Adair 456 times"
???
It's broken up into 12 categories...

Top Movies
Reports the top 10 movies of all time, the past year, month, or week. You can sort by rating, downloads, or views.

Top Reviews
Reports the top 10 reviews of all time, the past year, month, or week. You can sort by replies or views.

It also reports a list of all the people who have rated 10 or more movies, and what their average rating is.

Top Modifications
Reports the top 10 mods of all time, the past year, month, or week. You can sort by rating, downloads, or views.

New Threads
Reports the total number of new threads posted for each month, going all the way back to January 2006.

Most Active Threads
Reports the most active threads of all time, the past year, month, or week. You can sort by replies or views. You can also filter for a specific forum or all forums.

New Posts
Exactly like the "New Threads" report, except it is for posts instead of threads.

Most Active Posters
Reports a list of the most active users (determined by their post count) of all time, the past year, month, or week. You can filter for a specific forum or all forums.

Active Users
Reports a list of all of the users who have logged into their account at least once in the past 60 days. It also reports when they last posted and when they last logged in (unless they set their user CP to make them invisible on the who's online list).

Banned Users
This is just another banned users report like the one we already have.

User Activity
Reports the post count in each forum for individual users. Filter options similar to the reports above can also be applied.

User Ages
Splits the user base into age groups and counts the number of users in each of them. You can also filter for active users, inactive users, or all users.

It also reports the top 10 oldest and youngest users. You have to make your age public in your profile though in order to be included in these top 10s.

User Nationalities
Exactly the same as user ages, except it groups by the nationality flag you have set in your profile. You can also sort by all users, active users, or inactive users.

Quote:
Originally Posted by Kevitus
Come on you could turn it on for one day at least so we can see if it is really that bandwidth consuming.
It's not bandwidth that is the problem. It is CPU and memory usage on the MySQL server that I am concerned about. Some of the reports seem to cause severe spikes, and I've seen the whole site go down temporarily because of it. The "Most Active Posters" report in particular seems to be the worst, because it's basically running a SELECT query on the entire vbulletin_posts table, which has a half a million records in it now. I tried adding a cache mechanism so that after the report is run the first time, it displays a cached copy of the results rather than re-running the query. The cache expires the following day. Even with that, I've still managed to bring the site down by running the report just once during peak hours.

The "security" issues I have always sited are my concerns that a malicious user could come in and just start hammering one of the resource expensive queries until it overwhelms the server and causes a denial of service for other users. The caching mechanism was designed to prevent this, but I haven't got around to adding it to all the reports that need it.
Space Goat is offline   Reply With Quote
Old 05-31-2010, 01:54 PM   #108
bl0ppin
Senior Member
bl0ppin's Avatar
Join Date: Apr 2005
Posts: 4,644
Do it then.


bl0ppin is offline   Reply With Quote
Old 05-31-2010, 04:16 PM   #109
Phil Williamson
Super Moderator
Phil Williamson's Avatar
Join Date: Oct 2002
Posts: 16,900
yeah please do, that all sounds awesome


Phil Williamson is offline   Reply With Quote
Old 05-31-2010, 08:29 PM   #110
Kevitus
Senior Member
Kevitus's Avatar
Join Date: Jan 2003
Posts: 4,542
Quote:
Originally Posted by Space Goat
It's not bandwidth that is the problem. It is CPU and memory usage on the MySQL server that I am concerned about. Some of the reports seem to cause severe spikes, and I've seen the whole site go down temporarily because of it. The "Most Active Posters" report in particular seems to be the worst, because it's basically running a SELECT query on the entire vbulletin_posts table, which has a half a million records in it now. I tried adding a cache mechanism so that after the report is run the first time, it displays a cached copy of the results rather than re-running the query. The cache expires the following day. Even with that, I've still managed to bring the site down by running the report just once during peak hours.

The "security" issues I have always sited are my concerns that a malicious user could come in and just start hammering one of the resource expensive queries until it overwhelms the server and causes a denial of service for other users. The caching mechanism was designed to prevent this, but I haven't got around to adding it to all the reports that need it.

You could cronjob the statistics every week and then save the results or something so everything is only executed once a week.


Kevitus is offline   Reply With Quote
Old 06-01-2010, 05:07 PM   #111
Slime
Senior Member
Slime's Avatar
Join Date: Oct 2001
Posts: 24,891
I'm confused. I don't know anything about programming, but:

Shouldn't this thing just have stats that are fetched weekly, rather than on the spot? And then you could have it fetch them at a slowed pace rather than how it does it currently, crashing the server because its trying to fetch them quickly enough for the person that entered the search query?
If you just did them weekly, or twice weekly, it wouldn't have to fetch them quickly, AND you could do much more detailed stats because it wouldn't be overloading the server.

I just don't think anybody cares about having stats that pertain to THIS EXACT MINUTE, it would just be cool to have stats updated weekly or so.
I'd personally rather have more detailed stats than live stats.
Slime is offline   Reply With Quote
Old 06-01-2010, 06:43 PM   #112
Phil Williamson
Super Moderator
Phil Williamson's Avatar
Join Date: Oct 2002
Posts: 16,900
I'll take anything at this point


Phil Williamson is offline   Reply With Quote
Old 06-01-2010, 10:28 PM   #113
Space Goat
Administrator
Space Goat's Avatar
Join Date: Sep 2001
Posts: 6,042
I think you guys are missing the point.

It does not matter when or how often the statistics are fetched, it is where they are fetched from that is the problem. Setting it up to run weekly won't solve anything, because running some of the reports just once is all it takes to bring the server down. Whether you run it on demand or as a scheduled job makes no difference, it still has to run.

The only real way around the problem is to create a statistics table in the database, and populate it as users post and do other activities on the forum that we're interested in deriving metrics from. However, this is a lot more work, and wouldn't even likely be retroactive. I think that would suck and make much of it a lot less meaningful.

The only ones that I'm concerned about is the "Most Active Posters" and "User Activity" reports. I suppose I could just disable those for now, but I personally thought these were two of the most interested reports on the page.
Space Goat is offline   Reply With Quote
Old 06-01-2010, 10:55 PM   #114
Slime
Senior Member
Slime's Avatar
Join Date: Oct 2001
Posts: 24,891
And that's why I asked you if its impossible to just have it run slower, doing 1 page at a time since it has an entire week to do so. I figured that would be possible to program since I see PC programs do stuff like this (telling them to index files slower so that you can continue using your PC), but I don't know about web programming.
Slime is offline   Reply With Quote
Old 06-01-2010, 11:22 PM   #115
Space Goat
Administrator
Space Goat's Avatar
Join Date: Sep 2001
Posts: 6,042
No, that's not really an option. The queries in question don't lend themselves to being broken up into pieces. It needs to examine the entire table to get the total post counts.

Funny thing though is that I'm running them right now, and everything seems to be fairly quick. It might have just been the server was unstable last time I was testing this stuff, and just the little bit of extra load was pushing it over the edge.
Space Goat is offline   Reply With Quote
Old 06-01-2010, 11:50 PM   #116
Space Goat
Administrator
Space Goat's Avatar
Join Date: Sep 2001
Posts: 6,042
OK, the stats page is now live, minus the two reports that were bothering me. Enjoy.
Space Goat is offline   Reply With Quote
Old 06-01-2010, 11:54 PM   #117
Nixon
Senior Member
Nixon's Avatar
Join Date: Aug 2008
Posts: 4,293
Quote:
Originally Posted by Space Goat
OK, the stats page is now live, minus the two reports that were bothering me. Enjoy.
* pats on the back *


Nixon is offline   Reply With Quote
Old 06-02-2010, 12:46 AM   #118
Tuna Hematoma
Senior Member
Tuna Hematoma's Avatar
Join Date: Oct 2001
Posts: 10,055
I need to rate 4 more movies to get 600!


Tuna Hematoma is offline   Reply With Quote
Old 06-02-2010, 12:59 AM   #119
Slime
Senior Member
Slime's Avatar
Join Date: Oct 2001
Posts: 24,891
Neat, I like the Ages and Nationalities section.
Slime is offline   Reply With Quote
Old 06-02-2010, 01:03 AM   #120
Phil Williamson
Super Moderator
Phil Williamson's Avatar
Join Date: Oct 2002
Posts: 16,900
wheeee how fun thanks! Tuna and I give almost the exact same average rating!!!


Phil Williamson is offline   Reply With Quote
Old 06-02-2010, 01:22 AM   #121
Slime
Senior Member
Slime's Avatar
Join Date: Oct 2001
Posts: 24,891
oh cool, i didn't see that. i give a pretty low rating, wow. im surprised cause ive voted quite a few silly movies 100%
Slime is offline   Reply With Quote
Old 06-02-2010, 01:38 AM   #122
Nixon
Senior Member
Nixon's Avatar
Join Date: Aug 2008
Posts: 4,293
Quote:
Originally Posted by Tuna Hematoma
I need to rate 4 more movies to get 600!
you can go and rate all my movies 100% if you want, i dont mind


Nixon is offline   Reply With Quote
Old 06-02-2010, 01:51 AM   #123
James McCloud
Senior Member
James McCloud's Avatar
Join Date: May 2005
Posts: 4,214
or 0%.

both good choices.


James McCloud is offline   Reply With Quote
Old 06-02-2010, 02:20 AM   #124
Nixon
Senior Member
Nixon's Avatar
Join Date: Aug 2008
Posts: 4,293
Quote:
Originally Posted by James McCloud
or 0%.

both good choices.
hahaha your one funny guy!.........


Nixon is offline   Reply With Quote
Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


Sig Police

Contact Us | RSS Feed | Top

Powered By ezboard Ver. 5.2
Copyright ©1999-2000 ezboard, Inc.
Powered by vBulletin Version 3.5.3
Copyright ©2000 - 2022, Jelsoft Enterprises Ltd.