Operation Self Defense Needs coverage, I've got a fairly easy method

B

braindead0

New member
Short synopsis (so you don't have to read the whole thing): Operation Self Defense is trying to find news coverage of self defense uses of firearms. The sheer quantity of possible sources makes it difficult, so they get volunteers to search specific sites. This is pretty inefficient, I think I've found a better way for unix/linux/possibly mac.

========================
Any other folks with *nix servers, or just *nix boxes with network access?

I just finished, setting up my system to index 5 uncovered news sites, and email me an html page of search results on a list of gun related keywords every day (just previous day's news of course, can't index until it's posted). This should be much more efficient than going to individual sites and running several searches.

This method may work for Mac users, but I doubt windoze as HT://dig isn't going to compile there.

In addition, I can setup my system to do the same for another block of sites and mail the results to another reviewer. Not sure how many I can handle yet.

If we could find some more *nix systems to index and mail search results out to OpSD reviewers, I think we should be able to cover all the news sources eventually. There will likely be sites that we cannot index due to the way they are setup but I've had no problem so far.

If anybody has the hardware resources, and time.. PM me so that we don't clutter of general discussion with tech talk. I still need to clean up the scripts and setup examples in order for this to be really user friendly.

Pre-requisites

  • ht://dig
  • sendmail capable of forwarding to external addresses (or internal .forward capabilities?)
  • hd space (57megs for the 5 sites I'm doing)
  • some technical knowledge, shouldn't require an engineer though ;-)
 
B

braindead0

New member
Are there no *nix sysadmins out there?

I've been indexing and searching 5 sites daily, and emialing myself the search results...it works great! Takes about 1.5hours on my machine to do the index. It'll be faster onces I get all the crud outta the index (classifieds for example).

If there is anybody else that would like to participate in Operation Self Defense but doesn't want to go cruising to each site and searching manually, let me know. I could easily do this for at least one more person.

If I could just find a couple of *nix sysadmins that could donate some cpu/bandwidth to the cause, we could search a lot of sites with very few people...
 
B

BogBabe

New member
In a couple of months I'll be running Unix again on my Mac. I'm no Unix expert -- I know just enough to be dangerous, in fact.

But if you're willing to walk me through the steps in getting it set up, I'd be more than happy to help. (I could actually get it set up now; I just won't be running Unix on a regular basis until January -- long story, don't ask.)
 
B

braindead0

New member
Excellent! I have no idea of the capabilities of Mac os 10 (presumably what you'll be running) but it should be good to go.

The only tough thing I had to do was reconfigure sendmail with smarthost support so that I could use the local sendmail to send messages outside my local network. It wasn't too tough.

I'm going to clean up the scripts and consolidate them this weekend. I hope to make configuration and adding sites/users much easier.
 
B

BogBabe

New member
I forget which flavor of Unix, but OSX is completely Unix-based. I loved it while I was running it. I ran Apache, configured sendmail, created virtual servers, set up cron commands ... Had no real need to, but wanted to put it through its paces.

I've never done anything like what you're talking about though, so I'll need the step-by-step to get that going.

Are there no other Unix admins here?
 
B

braindead0

New member
Well, if you're going to do this just for yourself.. it probably makes sense to simply send the search results out to a file that you can open up and scan through on a daily basis. That way you don't have to mess around with sendmail configuration.

Once you get Mac OS X setup, probably the first thing to do is get a copy of ht://dig (if it's not already installed).

After I get done cleaning up these scripts I hope that it'll be very easy to setup and use....
 
B

braindead0

New member
I finally finished cleaning up my OPSD search scripts, I suspect it needs more documentation and probably some cleanup. In addition, the current version of ht://dig (the actual search engine) doesn't do phrase searching, so I get a lot of false hits from sporting articles ;-).

So far so good though. It takes about 1hr for my old, cheesy Celeron to index and search 5 newspapaper sites. I'll be taking on another 5 I think.

I would attach a copy, but I can't ;-).
Likely hosts for this would be Unix/Linux servers or workstations with internet connectivity. If you know of any system administrators that might be interested, feel free to pass this on.

Here's a link to the file on my server
 
You must log in or register to reply here.
Top