Scoop -- the swiss army chainsaw of content management
Front Page · Everything · News · Code · Help! · Wishlist · Project · Scoop Sites · Dev Notes · Latest CVS changes · Development Activities
RFC: Scoop Archiving Feature Requests
By Captain_Tenille , Section Project []
Posted on Mon Sep 24, 2001 at 12:00:00 PM PST
Here's my Request for Comment for archiving of older stories in Scoop. Please feel free to comment with suggestions, both for features you'd like to see implemented and ones you think should be removed from the plan. I haven't started coding yet, so this document is very flexible.

What the Scoop archiving process should do is provide a way to take a load of the database server by moving the intro, body, and comments from old stories out of the database and into a flat file. However, it should also be done in such a way that dynamic boxes are still displayed, and comments can possibly still be posted.

My ideas on this are as follows:

First: the actual archiving process.

This should be run as a cron job that will search the database for any stories that are older than the cut-off time for archiving (set as a var, default three weeks). The job will go through these stories, slurp up their intros, bodies, and comments (which should be archived nested, I think), and write them to a file, which will be stored in the filesystem something like /usr/local/www/scoop/archive/2001/9/24/343453/23424/intro.html. Polls will be left alone. The job will then change the stories display status to "4", thus telling Scoop that the story has been archived. Finally, the intro, body, and comments are deleted from the database.

Second: displaying archived stories.

Stories.pm (and any other appropriate files) should have a conditional added to check if a story has been archived. If it has, instead of querying the db for the intro, body, and comments, it will slurp them out of the file (looking in the directory that looks like the story's sid) and put them into one giant |CONTENT|in the story. The rest of the page will use the standard story_template.

Third: posting comments and rearchiving

On the off chance that someone wants to post a comment to a story that's six months old, the "Post a Comment" link will remain. If somebody posts a comment to an archived story, the story will get put back in the database, the comment will get posted, and the story remains in the database until it's been inactive long enough for it to be archived again. To facilitate putting the comments back in, there should be markers placed in the comments file to help put everything back where it should be.

Other Stuff

There needs to be a var to set how often stories will be archived. Also, it should be possible for admins to either deny anyone permission to post in an old story, or restrict it to users with certain permissions. There is also a question about what should happen to 0 rated posts.

Any suggestions, ideas, thoughts on how the best way to integrate this into the codebase, or usability are more than welcome. If there is no way this would work with Scoop the way it's outlined, or anyone can think of a better way, please tell me that too. :-)

-jeremy, aka "Captain_Tenille"

< ids.org.au and anime.org.au | Usability guff: label tags for poll options >

Menu
· create account
· faq
· search
· report bugs
· Scoop Administrators Guide
· Scoop Box Exchange

Login
Make a new account
Username:
Password:

Related Links
· Scoop
· More on Feature Requests
· Also by Captain_Tenille

Story Views
  48 Scoop users have viewed this story.

Display: Sort:
RFC: Scoop Archiving | 3 comments (3 topical, 0 hidden)
disagree (none / 0) (#1)
by eries2 on Tue Sep 25, 2001 at 09:50:01 AM PST

I think this is not a bad idea, but I would suggest taking a look at Rusty's ideas that I've seen around here and K5 that provide for something similar, except it actually writes to disk a serial representation of the in-memory perl object that encapsulates the stories. That way it is still possible to do some dynamic stuff on the data without touching the database.

Just my $.02

Eric



Archving comments and Searching. (none / 0) (#3)
by hurstdog on Tue Sep 25, 2001 at 10:29:38 AM PST

I'm wondering how we would archive comments. Should they be taken out of the database as well? Or should they be left out of this archived page with just placeholders, like |comment_<num>|? If they are pulled out of the database as well, this could cause significant strain on the database when someone posts a new story. Scoop would have to reload the 2 parts to the story, as well as the comments, back into the database. That could easily be a Megabyte of data insertion.

After thinking about reloading all of that data, I think I understand what Eric is talking about below. We wouldn't have to just reload the intro, and the body, but also the rest of the fields from the database. Hmm... of course, we don't have to take those fields out. Ok, I think it would work best if just the body and the intro were cached, since thats the main part of the story that gets big. Though you might have already wanted to implement it that way. :) That way Stories By Author searches would still work.

But if we were going to store all of the comments.. it might be better to leave them as objects, or one big object in a file. That way we can get to them with $comments->[<num>]->{text}. We could even leave most of the comment in the database too, like with the story. Just take out the sizeable part, the comment text.

Which brings up another thing. If we take out the story body/intro, and the comments from the database, how do we search them? Would we take a sampling of the words from the story, and leave them in the bodytext field so that the search would find them? How would we tell which words to save and which to ignore? Searching could get really hairy really fast, I would think. Especially if we tried to search all of the flat files, of the archived stories. And even more so if they were stored as objects...

Hmm.. something to ponder.



-hurstdog


RFC: Scoop Archiving | 3 comments (3 topical, 0 hidden)
Display: Sort:

Hosted by ScoopHost.com Powered by Scoop
All trademarks and copyrights on this page are owned by their respective companies. Comments are owned by the Poster. The Rest © 1999 The Management

create account | faq | search